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FOREWORD 



The National Forum on Education Statistics (the Forum) is pleased to present Traveling 
Through Time: The Forum Guide to Longitudinal Data Systems. This document, Book Four 
of Four: Advanced LDS Usage, is the fourth and final installment of this Forum series of 
guides on longitudinal data systems (LDS). One goal of the Fomm is to improve the 
quality of education data gathered for use by policymakers and program decisionmakers. 
An approach to furthering this goal has been to pool the collective experiences of Fomm 
members to produce “best practice” guides in areas of high interest to those who collect, 
maintain, and use data about elementary and secondary education. Developing LDSs 
is one of those high-interest areas. These systems hold promise for enhancing both the 
way education agencies use data to serve students and the way they do business, from the 
policy level to the school office and into the classroom. 

LDSs are increasingly becoming the state of the art in education data. These systems 
move us from relying on blunt, aggregate, snapshot student data; to detailed and timely, 
student-level data that reflect the student’s entire academic history. An LDS makes it 
possible to not only monitor the success of individual students, but also to identify trends 
in those students’ education records. Freeing educators from guesswork and lessening the 
burden of painstaking data analysis, these systems provide powerful and timely insight 
about students and allow educators to tailor instruction to better meet individual needs. 

An LDS can reveal with great clarity what effects our policies, programs, and decisions have 
on schools. These systems allow agencies to track students across institutions to facilitate 
appropriate course placement and to determine who has transferred and who has dropped 
out. Longitudinal data systems also offer a new level of sophistication at the business 
level that can streamline operations; improve data quality; and free up valuable resources 
previously allocated to inefficient data entry, maintenance, and reporting practices. 

For these and others reasons, states should continue to introduce, develop, and 
expand their LDSs. The Traveling Through Time: The Forum Guide to Longitudinal Data 
Systems series is intended to help state and local education agencies meet the many 
challenges involved in developing robust systems, populating them with quality data, 
and using this new information to improve the education system. The series introduces 
important topics, offers best practices when possible, and directs the reader to additional 
resources related to the LDS development process. In sum, it is intended to help agencies 
establish LDSs that will have lasting, far-reaching impact on the education system and on 
students’ lives. For a description of the entire guide series, see appendix A. 



Book Four of Four: Advanced LDS Usage 

This fourth book in the guide series focuses on issues important to the effective use of 
longitudinal data. 

> The introduction explains the purpose of book four, as well as the format and 
conventions used throughout the series. 

> Chapter 1 takes a historical look at data use, comparing what was done in the past 
with what is possible today thanks to detailed longitudinal data. 

> Chapter 2 describes several preconditions to effective data use. 
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> Chapter 3 summarizes the many uses of longitudinal data, and the types of users 
who can leverage the information. 

> Chapter 4 offers tips on implementing effective training and professional 
development programs on the use of longitudinal data. 

> Chapter 5 discusses the road ahead for LDS development and use. 

The appendices include an overview of the four books in this series, references, 
additional resources, and other relevant Forum and NCES resources. 

The National Forum on Education Statistics 

The work of the Fomm is a key aspect of the National Cooperative Education Statistics 
System (the Cooperative System). The Cooperative System was established to produce 
and maintain, with the cooperation of the states, comparable and uniform educational 
information and data that are useful for policymaking at the federal, state, and local levels. 
To assist in meeting this goal, the National Center for Education Statistics (NCES), within 
the U.S. Department of Education, established the National Forum on Education Statistics 
(the Forum) to improve the collection, reporting, and use of elementary and secondary 
education statistics. The Fomm deals with issues in education data policy, sponsors 
innovations in data collection and reporting, and provides technical assistance to improve 
state and local data systems. 

Development of Forum Products 

Members of the Fomm establish task forces to develop best-practice guides in data- 
related areas of interest to federal, state, and local education agencies. They are assisted 
in this work by NCES, but the content comes from the collective experience of the state 
and school district task force members who review all products iteratively throughout 
the development process. Documents prepared, reviewed, and approved by task force 
members undergo a formal public review. This public review consists of focus groups with 
representatives of the product’s intended audience, review sessions at relevant regional 
or national conferences, or technical reviews by acknowledged experts in the field. In 
addition, all draft documents are posted on the Forum website prior to publication so 
that any interested individuals or organizations can provide feedback. After the task force 
oversees the integration of public review comments and reviews the document a final time, 
publications are subject to examination by members of the Fomm standing committee 
sponsoring the project. Finally, the entire Forum (approximately 120 members) reviews and 
formally votes to approve all documents prior to publication. NCES provides final review 
and approval prior to publication. 
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INTRODUCTION 




in 

he three preceding books of this series covered the groundwork for 

successful longitudinal data systems (LDS), creating a foundation of good 
data governance; envisioning the desired system with broad stakeholder involvement; 
developing a robust and sustainable system that collects and maintains valuable new data; 
ensuring data quality by training staff, enforcing the data governance process, and adhering 
to data standards; and securing the data and protecting privacy. But, as with any useful 
tool, building an LDS is not enough: the system and its data must be put to good use. The 
vast new inventory of longitudinal, student-level data education agencies will amass should 
be used frequently and effectively to help policymakers determine what works and ensure 
investment in the right educational inputs; help educators target instruction to improve 
student outcomes; give students and parents access to valuable, real-time information; 
and streamline agency and school operations. This book focuses on the ways the data can 
be used by stakeholders, as well as the tools and training that will allow users to access 
and turn student-level longitudinal data into actionable information at all levels of the 
education system. 

Figure 1 lays out the major issues discussed in each of the four books in this series. 
For more information on the purpose, format, and intended audience groups of this guide 
series, see Book One: What is an LDS? 
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Figure 1 . Summary of the Forum Guide to LDS series 



Book hWhot Is on LDS? Book II: Planning and 

Developing on LDS 



■ Understanding what on LDS is (and is not) 

■ Appreciating the organizational steps needed to 
institute and effectively use an LDS 

■ Identifying the technical features and 
capabilities of an effective LDS and the 
additional features that can enhance the 
system's utility 

■ Recognizing the benefits of an LDS 



■ Engaging stakeholders 

■ Describing the current system 

■ Envisioning the desired system 

■ Defining needs, including data and functionality 

■ Gaining buy-in and funding 

■ Building relationships 

■ Writing an RFP 

■ Building or buying a system or components 

■ Transferring knowledge (e.g., from developers 
to staff) 




■ Defining and measuring success 

■ Refining the system 



Book III: Effectively 


Book IV: Advanced 


Managing LDS Data 


LDS Usage 


■ Defining governance structure 


■ Collecting, storing, and delivering key data 


■ Defining roles and responsibilities 


■ Developing useful reports to fulfill common 


■ Collaborating to improve data quality and 


data requests and needs 


streamline operations 


■ Developing user-friendly data tools to facilitate 


■ Managing changes to the system 


access and analysis 


■ Training staff to ensure data quality 


■ Training users to utilize the technology 


■ Auditing/validafing data at all levels 


■ Building awareness, understanding, and 


■ Establishing/following data standards 

■ Securing data to protect privacy 

■ Providing users access to key data 


analytical capacity 
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Chapter 1 



DATA USE: WHERE HAS IT BEEN, 

Where IS it Going? 




I hile education agencies have long been collecting large quantities of data, 

I these usually aggregated, group-level data have mostly been used to meet 
state and federal reporting requirements for funding and accountability purposes, and to 
periodically provide blurry, high-level glimpses of the reality on the ground. Data use to 
improve education services at both the policymaking and classroom levels, on the other 
hand, has been rare. 



Data Use: More than just IT 

I f you implement it, they may come.. .but will they like what they find? As stressed 
earlier in this guide series, an IDS should not simply be an IT project, and data 
use is the reason why. While technical staff will develop the system, an IDS should 
be designed by and for the end users if it is to meet their needs. 

The IT staff can translate the business needs into a system that includes 
all the specified data elements. They can follow the defined business rules 
as requested and validate the data; and, in the end, they can create a system 
that "works" in a technical sense. But ultimately, the end user may find the 
system lacking if it is merely a collection of data and government reports. And, 
without useful tools and stakeholders who understand how to use these data 
in a meaningful way, the IDS will likely be under-utilized and fall short of its 
transformative potential. In a sense, it could end up being just an IT project after 
all. To avoid this, the system's design and development should be focused on using 
data not only to improve operations, but to ensure greater student success. 
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Many factors have contributed to this reality. Educators have lacked data that 
were timely and granular enough to be very useful at the classroom and student levels. 
High-level snapshot data were often available, but educators and researchers did not 
have access to the detailed longitudinal data needed for deeper and more meaningful 
analysis. Motivated educators hungry for data found accessing and manipulating the 
information difficult as it required collecting data from disparate sources and points 
in time, then painstakingly analyzing them by hand or through other time-consuming 
means. In general, educators have not been taught how to use data effectively (during 
pre- or in-service teacher preparation programs), largely because institutional cultures 
simply have not valued and encouraged data use (Wayman 2005). Education agencies 
have made laudable advancements in recent years, developing LDSs with many or all of 
the major components and functionalities. Yet these systems have often failed to lead to 
the effective use of the new student-level, longitudinal data; and, thus, to substantially 
benefit the full range of potential stakeholders. 

But the landscape is quickly changing. At both the state and local levels, carefully 
planned, skillfully developed LDSs— in combination with user-friendly tools and strong 
cultures of data use— are moving the education community into an era rife with objective 
inquiry. Agencies are beginning to transcend the historical purposes of data collection, 
empowering stakeholders throughout the education community by expanding their use 
of data to improve their work and student outcomes. They are increasingly using data 
to inform decisions about everything from policymaking and resource allocation, to 
professional development programming and targeting, to instmctional planning and 
intervention at the classroom and student levels. As increasingly sophisticated LDSs are 
implemented and expanded, and greater use of data is fostered among stakeholders, major 
changes are occurring. 



Data: From Local Burden to Universal Resource 




With a sophisticated 
LDS, data become a 
universal resource, 
not just a local chore 



Historically, many local education staff might have summed up their relationship with 
data as “we make them, they use them.” That is, school and district staff collected and 
reported data to comply with state and federal requirements and receive funding, but rarely 
used them to enhance their own work. Making data locally available in a useful and timely 
manner was difficult due to technological challenges, and often was not a priority. With 
the advent of robust and accessible LDSs, however, the local data burden is increasingly 
leading to greater reward."' When useful data are made available to local stakeholders, 
the dynamic shifts to “we make them, everybody uses them.” from the local perspective, 
this is a fundamental repurposing of education data from an administrative requirement 
to a valuable resource— an informational asset that empowers stakeholders at all levels to 
improve students’ educational outcomes. Indeed, data use is growing among educators and, 
already, many administrators and teachers no longer make significant decisions without 
objective, high quality facts in front of them. 



’^Compounding this change is the decrease in burden made possible by interoperability solutions that reduce data entry 
errors and eliminate the need for redundant data entry into multiple systems. 
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Policymaking and Administration: From Guesswork to 
Evidence-Based Decisionmaking 

Historical use of aggregate data to gauge performance, evaluate success, and plan for 
improvement is slowly but surely giving way to the use of timely, detailed data that offer 
substantially greater precision and reliability. Armed with more granular, accurate facts 
about what is happening on the ground, policymakers and school leaders can better assess 
what is working and what is not. In effect, this enables them to allocate resources more 
efficiently and effectively. For instance, decisionmakers can more objectively and reliably 
gauge the real impact of policies and programs on student outcomes. And, by linking 
and aggregating longitudinal, student-level data to various educational inputs, leaders can 
determine which schools are most effectively teaching which subjects to which types of 
students (for example, those who have struggled academically with particular concepts and 
skills, etc.). Professional development needs are also more easily identified and targeted 
to ensure all educators are well-qualified. In effect, leaders can trade hunches for facts, 
supposition for the scientific method. 




Teaching: From Intuition, Isolation, and Rigidity to 
Objectivity, Collaboration, and Fluidity 

Teaching has traditionally been a private matter, largely dependent upon the instructor’s 
intuition and skills. Today, with detailed, relevant, and timely student-level data accessible 
through user-friendly tools and reports, as well as social networking processes and tools 
(online bulletin boards, professional learning communities, and other resources), teaching 
is quickly becoming a more objective and open endeavor. The reality is, no teacher can 
discern the patterns in every student’s performance and behavior, much less figure out 
how to meet individual needs most effectively— especially when they are responsible for 
a large number of pupils. Better data and analytical tools can clarify reality, simplify the 
information teachers have to process, reveal patterns in performance and the effects of 
their own actions, and remove biases from interpretation. And with more information 
available, more and more educators can collaborate to learn from the data, reflect on 
experiences, and share effective practices and instmctional materials as evidenced by the 
data. Reliance on infrequent, summative tests is fading in favor of frequent, interim and 
formative assessments that measure growth, pinpoint deficiencies, and help teachers adjust 
instruction throughout the school year (see the “Balanced Assessment System” section in 
chapter 3 for more information). With the new insights revealed by the data, educators are 
better able to assess their methodology, differentiate instmction for groups and individual 
students, monitor student progress throughout the year, and continually make adjustments 
to better meet students’ needs. 



Book Four of Four: Advanced IDS Usage 



7 




Teaching in this century 

T he school year had just begun at Century Elementary School in Grafton, North Dakota. 

Children in this K-4 building had already taken assessments in math and language arts. 
Results were quickly distributed to staff and, based on fhe findings, pupils were placed in 
core and fier insfruction groups wifh similarly scoring peers. 

Throughouf fhe year, fhe sfudents' performance is confinually and individually monitored for 
progress. Teachers meef on a regular basis wifh fhe school's response to intervention (RTI) 
coach, guidance counselor, and principal to adapt instruction to students' advancements and 
changing needs. 

"Teachers today must collaborate and work as a team using timely data," says the principal. 
"School now follows a fluid model." 



Data Sharing: From Partitions to Partnerships 

Data sharing among institutions was limited in the past and continues to pose a host of 
logistical challenges. Today, data sharing is a growing reality among schools as well as 
districts. In addition, K-12 education agencies share data with early childhood and 
postsecondary education, and other agencies such as labor and social services. Unique 
identifiers and improved communication among and across levels of the education system 
are providing a clearer picture of where students are, how they are progressing, and what 
their long-term outcomes are; as well as a detailed view of how their experiences relate to 
their success. For instance, what effects might early childhood education have on 
performance in elementary school? Elementary years on middle school? Middle on high 
school? High school on postsecondary education and workforce success? And for students 
who go on to be teachers, what effects might pre-service training have on their future pupils’ 
performance? Data sharing throughout the P-20 system can help provide the answers to 
these important questions. 

Asking the data: Then and now 

Longitudinal, student-level data help put reality under a microscope in order to understand 
the details of what is going on. This allows much more precise analysis of the effects of 
policymakers’ and educators’ actions, as well as other factors that contribute to educational 
success or failure. (For the types of questions these data can answer, see chapter 5 of Book 
One: What is an LDS?) 
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Table 1 illustrates the transformation enabled by student-level longitudinal data. 

The before-and-after presentation compares the types of questions traditional, aggregate 
snapshot data can answer about the education system (questions without an LDS); to the 
more detailed information available with an LDS. As the table shows, the data offer a 
better understanding of what is actually happening in schools. The questions in this table, 
taken directly from Information Won’t Be Used If No One Can See It (Dougherty 2008), are 
policy-level questions that rely on summative assessment data. More detailed questions can 
be answered with more detailed data, such as formative assessment data, input data, and 
other information. 



Table 1 . Policymaker questions: Before and after development of o sophisticated LDS 



Data use 


Questions without an LDS 


Questions with an LDS 


Analyze 

assessment 

results 


■ How many students were proficient in 8th 
grade? 


■ Disaggregated by prior achievement: How 
many students who were proficient in 8th 
grade stayed proficient in 10th grade? 




■ How many students scored below basic in 4th 
grade? 


■ What percentage of below-basic 4th graders 
normally reach proficiency by 8th grade? 




■ What percentage of this year's high school 
seniors took at least one Advanced Placement 
(AP) course and passed the corresponding AP 
exam? 


■ How is completing an AP course and passing 
the AP exam related to students' 8th grade 
achievement levels? 




■ Which schools and districts hove the highest 
test scores within their demographic category? 


■ Which schools and districts do the best job 
of producing growth among academically 
advanced students? Are these different from 
the schools and districts that do the best job 
of helping students who ore academically 
behind catch up? 


Analyze 

graduation 

rates 


■ How many students graduated this year? 


■ What percentage of students who completed 
8th grade four years ago graduated on time 
this year, taking into account transfers and 
dropouts? 




■ How does the number of 1 2fh graders in 
each high school compare to the number 
of 9th graders three years earlier? 


■ What causes these differences, taking into 
account transfers, dropouts, and students held 
back a grade? 




■ Does the number of graduates relative to 8th 
graders in this district appear to be declining? 
Were 8th grade test scores increasing or 
declining four years ago? 


■ How are students' graduation, dropout, 
and transfer rates related to academic 
performance levels in 8fh grade; and how are 
those relationships changing over time? 
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Analyze course 

completion 

data 


■ What percentage of sfudenfs in each grade 
received course credif for algebra II? 


■ What percentage of sfudenfs receiving course 
credif for algebra II met college readiness 
benchmarks on the ACT* or earned a 
corresponding score on the SAT*? 

■ What percentage of sfudenfs who complefed 
algebra II also demonsfrafed proficiency on an 
algebra II end-of-course exam? 




■ Whaf percenfage of high school juniors and 
seniors fook algebra II or courses beyond 
algebra II in fhe pasf two years? What 
percentage of sfudenfs in fhe stafe's colleges 
needed remediafion in mafhemafics fhis year? 


■ How many of lasf year's graduates who 
completed mathematics through at least 
algebra II in high school and entered college 
this year needed remediation in mathematics? 




■ Whaf were 8fh grade proficiency rafes for 
mafh fwo, fhree, and four years ago? 


■ How do the answers to the questions above 
vary for sfudenfs based on fheir mafhemafics 
performance level in 8fh grade? 


Compare K-12 
performance to 
postsecondary 


■ Whaf are fhe high school graduafion frends in 
fhis disfricf? Whaf are fhe college enrollmenf 
frends in fhe region and state? 


■ Whaf percentage of fhe sfudenfs from fhis 
disfricf enroll in higher education within two 
years of high school graduafion? 


success 


■ What are the trends in student grades, majors, 
and graduation in the two- and four-year 
colleges in fhe region and sfate? 

■ Whaf are fhe frends in fhe academic 
achievemenf levels and course-faking pafferns 
of fhis disfricf's high school sfudenfs? 


■ Whaf are fhe frends in college grades, majors, 
and graduafion for sfudenfs from fhis disfricf; 
and how are fhey relafed fo fhose sfudenfs' 8fh 
grade achievemenf levels, high school course- 
taking patterns, and high school academic 
achievement? 


Compare 
elementary 
and middle 
school 
performance 


■ Whaf are fhe achievemenf frends in fhis 
disfricf's elementary, middle, and high 
schools? 


■ How do students from fhis elementary school 
perform after they move on to middle school? 
How do these middle school students perform 
on sfate assessments and college readiness 
tests in high school? 


fo success in 
high school 


■ What are the course-taking patterns in this 
district's high schools? 


■ How many of fhe sfudenfs from fhis middle school 
fake rigorous courses in specific subjecfs in high 
school, and how do fhey do in fhose courses? 




■ What percentage of sfudenfs are performing af 
specific achievemenf levels in fhe middle and 
high schools fhaf sfudenfs leaving this school 
normally attend? 


■ How does fhe performance of fhis school's 
sfudenfs in fhe next level of educafion relafe to 
their academic achievement level when they 
leave this school? 



SOURCE: Dougherty, C. (2008) Information Won't Be Used If No One Can See It. 
Reprinted with permission of the Data Quality Campaign. 



“‘■American College Testing (ACT), Scholastic Assessment Test (SAT) 
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A Case Study; 
From Inches to 
YARDS 



Part I: The struggle to expand data use 



T he Cambridge Public Schools in Massachusetts had been 
working on its districtwide IDS for some time. Several years 
of sfudenf dafa were housed within the district's local data 
warehouse — demographics, enrollment, attendance, discipline, 
classroom grades, state and local assessment results, etc. But 
despite this rich collection of dafa, use of fhe informafion was 
limifed. In facf, only a few sfaff members in fhe enfire disfricf were capable of accessing fhe dafa fo frock down 
informafion, conducf analyses, and produce reports. And, aside from fulfilling reporfing requiremenfs, fhe disfricf 
had used fhe dafa for only two purposes. 



First, they produced annual reports that compared student outcomes disaggregated by demographics. This 
established benchmarks to help the community see how well the school system was achieving its goals. Then, 
the district provided teachers with historical data snapshots of each studenf fhey would have fhe following 
year — a resource educafors found very helpful in preparing for fhe new school year. 



Despite these two very productive data uses, district leaders knew that their inability to provide broader access to 
the system was a significant missed opportunity. They needed to provide administrators, teachers, researchers, 
parents, students, the public, and city programs and services with greater access to these valuable data. The 
district had a history with commercial technology solutions, but its experience suggested that, rooted in the 
business world, these products were inadequate to meet the unique needs of fhe education community. In spite 
of fhis, fhe disfricf again fumed ifs sights outward for help. 



Over fhe nexf year or so, fhe disfricf fried fo defermine how fhey could more effecfively use fheir dafa. They held 
meefings wifh various sfakeholders and came up wifh a lisf of five main indicafors of sfudenf success or failure fhaf 
fhey wanfed fo frock. They also knew fhey wanfed fo follow sfudenf cohorfs and explore fhe factors relafed fo fheir 
performance, and fo exfend analyfical funcfionalifies down fo fhe classroom level. Armed wifh a basic idea of whaf 
fhey wanfed, fhe disfricf released a requesf for informafion (RFI) fo see whaf solufions were available. They received 
over a dozen responses. Affer a few presenfafions, fhe sfaff culled ouf fheir favorife funcfionalifies from fhe vendors' 
recommendafions and craffed a defailed requesf for proposals (RFP) fo find fhe righf solufion. 



Affer hearing from fhe few respondenfs who said fhey could meef fhe requiremenfs, fhe disfricf chose a stort-up 
company whose suife of fechnology solufions was designed fo make educafion dafa more useful and accessible fo 
sfakeholders. The vendor promised if had solved many of fhe problems fhaf had plagued fhe educafion dafa field 
for years, and was capable of delivering wifhin a shorf fime frame a flexible solufion fhaf would meef fhe disfricf's 
currenf requiremenfs yef accommodafe fufure needs. 



Confidenf in fheir choice, disfricf leaders chose a number of modules from the vendor's suite that would provide a 
wide range of funcfionalifies: 

■ a dafa warehousing solufion would infegrafe and store data from various sources (e.g., stafe and 
local assessmenf dafa, dafa from other city agencies and services) and provide role-based access and 
extensive reporting and analysis capabilities down to the classroom level; 

■ an online testing platform with reporting capabilities would allow educators to easily craft online 
assessments tied to academic standards, and to then quickly pull up detailed analyses of fhe results; 

■ an instructional management system would put a host of resources af users' disposal to help educators 
create lesson plans and assign work to students; and 
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■ a social networking tool would connect students, parents, teachers, principals, and district administrators 
in a secure environment. With a look and feel similar to leading modern social media and networking 
tools, the application's familiarity would hopefully make the tool second nature to many users. 

These modules would be rolled out individually and accessible through a single, user-friendly interface where 
users would be able to view longitudinal data, gauge progress towards education goals, access testing and 
instructional resources, and easily communicate with one another. 

Now it was time to put the vendor and its solution to the test. The district moved forward with cautious optimism. 

Part II: Rollout, training, and a look to the future 

Rollout of the Youth and Resource Development system (YARDS), as the district's system was named, was planned 
in stages, with broad expansion of access to longitudinal data as priority number one. A speedy timeline was 
set for the initial phase and the vendor was able to quickly integrate ten years worth of longitudinal student data 
from various sources, correcting many data quality issues in the process. Within phase one, teachers, principals, 
and district administrators were also given access to customized dashboards to provide access to this wealth of 
information: 

■ District administrators received easy access to up-to-date information on districtwide operations and 
performance data, drillable down to the school, classroom, and student levels. 

■ School principals were given school and classroom overview data that can be disaggregated down to the 
classroom and student levels. 

■ Teachers gained easy access to classroom- and student-level information such as demographics, 
attendance, assessment performance, participation in programs offered by the district or other city 
agencies, and alerts about struggling students. Teachers can access these data during the summer to 
learn about the students they will soon have in class. Whereas previously teachers needed to request 
most data from the school office, the new software put even a wide variety of historical data at their 
fingertips. 

Through these dashboards, data coaches and other staff were able to conduct analyses at the aggregate level 
to help teachers and administrators monitor student outcomes, identify trends in cohorts, gauge the effects of 
programs, and determine how to boost learning and improve performance. School principals and administrators 
were able to generate and publish reports online to share with specific audiences; for example, reports on 
grade-level assessments might be shared and used to collaborate with teachers. In the next phase, teachers will 
use electronic grade books and report cards; and educators and students will gain access to goal planning and 
progress monitoring tools towards grades, graduation, higher education objectives, etc. 

Training 

Professional development efforts began before the system was in place. But rather than introduce the technology 
as the main issue of fraining, the district decided to address the broader issue of data analysis and introduced 
the technology as a part of the bigger picture — the tool rather than the focal point. At the high school level, 
training focused on administrators and teachers; for K-8, data and curriculum coaches were trained to work with 
teachers only. When the system is fully operational, staff will have fime built into their schedules for professional 
development so they may learn to use the software and conduct their own data explorations and analyses. 
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The future 

The district is very excited about the YARDS initiative. Whiie impiementation has suffered some setbacks, mostiy 
reiated to difficuities defining a districtwide vision for data uses, the district has been encouraged by its success 
so far and continues to activeiy pursue its goais for the new system. Leaders envision extensive use of data with 
the heip of their new toois. As new moduies are introduced, they hope to siowiy incorporate the technoiogy more 
and more into the schooi cuiture. Their uitimate goai is a significant transformation— a new environment where 
students, parents, teachers, and administrators can easiiy and intuitiveiy access data, instructionai materiais, and 
assessments; communicate with ease; and coiiaborate using data to improve education, 
in addition, district ieoders hope to expand data sharing and use weii beyond schooi waiis. The uitimate goai is 
for YARDS to become a citywide resource that connects a host of city-run organizations. This wouid enabie broad 
coiiaboration towards improved city programs and services, and better ensure the academic success and overaii 
weii-being of aii of Cambridge's students. 
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Chapter 2 



Prerequisites eor Eeeective 

DATA USE 



fa 

or an LDS to reach its potential, certain conditions of high quality data use 
must be met. These prerequisites fall into four basic categories: data, access, 
understanding, and support. Specifically, to exploit the information effectively, users must 
have easy access to, and an understanding of, high quality data; as well as strong leadership 
and organizational support. The more effectively agencies achieve these ideals, the more 
successfully the system will be utilized. Without effective realization of these conditions, 
the system is likely to fail in various predictable ways. 




To exploit the information effectively, users must have 
easy access to, and an understanding of, high quality 
data; as well as strong leadership and organizational 
support. 



To fully benefit from the LDS, stakeholders should have 

> high quality data that meet a broad range of user needs and are accurate, 
comparable and trustworthy; 

> easy access to key information (e.g., performance and evaluation indicators) 
through a web portal, reports, and analytical and querying tools; 

> an understanding of what data are available to them, how to use the tools to 
access and analyze the information they need, and how to make sense of the facts 
presented or interpret the results of their own analyses; and 

> support from strong leadership, colleagues, and peers who value, model, and 
facilitate effective data use. 

Figure 2 illustrates these requirements. 
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Figure 2. Data use framework 



Prerequisites for effective LDS use 


QUALITY 

DATA 

• 


■ Stakeholder engagement 

■ End user's needed data 

■ Strong data governance 

■ Adherence to common data standards 

1 ■ Linkages within and beyond K-12 


ACCESS 


■ Data warehouse (or linked data repositories) 

' ■ Web porta i 

■ Reports (standardized & customizable) 

■ Analytical and querying tools 


UNDERSTANDING 

• 


■ Awareness of available data 

■ Training (on use of technology) 

■ Professional development (on data use 
and intepretation) 


SUPPORT 


■ Leadership support 

1 ■ Organizational suport (e.g., small groups, 

support staff) 

■ Time reserved for analysis 



Quality Data 

In Book Two: Planning and Developing an LDS, the importance of defining and meeting 
stakeholder information and analytical needs was discussed. And in Book Three: Effectively 
Managing LDS Data, data quality was addressed in terms of ensuring accuracy, timeliness, 
validity, and utility. This foundation of high-value, high-quality data provides the raw 
materials for effective LDS use. 
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> Stakeholder engagement to envision ideal and gain buy-in 

System success begins with stakeholder interest, established through early 
and continual involvement. End users should be involved in the design and 
refinement stages, including the selection of data items that will be collected and 
made available through the system. 

> LDS POPULATED WITH VALUABLE, NEEDED DATA 

The system should be designed to include all data stakeholders need to answer 
their questions, including those needed for reporting requirements. This detailed 
information, including historical data, must be linked over time and cover areas 
such as student, staff, institutional, curricular, achievement and assessment 
(formative and summative), programmatic, and financial information. The 
system should be designed to answer known information needs, yet be flexible 
enough to accommodate newly identified requirements and new data items as 
needed. 



> Quality data ensured through a strong data governance 

PROCESS 

Decisions based on bad data are likely to be deficient. Therefore, a strong 
governance process and staff training program should be implemented to ensure 
data quality and utility. 

> Adherence to common standards 

Adherence to common data standards facilitates data quality, interoperability, 
and comparability across systems. Agencies should follow widely accepted data 
standards including definitions, code sets, technical specifications, and other 
guidelines. 

> Integration of K-12 agency data and linkages to outside 

DATA SETS 

Ideally, K-12 data systems should be linked or integrated with data from early 
childhood education, postsecondary, the workforce, neighboring states, and 
other agencies as determined by the state or district. This broader integration 
allows a more complete picture of students’ experiences throughout their 
education and professional lives. 



Access to the Information 

Beyond the crucial foundation of quality data, users must have the tools to access the 
information they need and are authorized to see. Without easy access, the information 
may require tedious work to find and decipher, making data use less likely. 




> Data warehouse (or linked repositories) 

Agency data should be consolidated in a central data warehouse (or linked across 
multiple source files) to allow users to easily yet securely access and analyze key 
information from different areas of the education system. Data from separate 
systems should be transferred frequently to ensure they are current. 
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> Web portal 

Data should be available via a user-friendly web portal that provides secure 
access to the agency’s reporting and querying tool. This single point of access can 
provide easy access to longitudinal reports and analytical tools from a wide range 
of sources. 

> Useful and timely reports 

Agencies should create pre-packaged, high-quality reports to answer frequently 
asked questions or provide key information. These reports should clearly and 
concisely communicate important information and give users context to ensure 
accurate and appropriate interpretation. They should also be timely to allow 
stakeholders to take action in the present rather than simply reflect on the past. 

> Analytical and querying tools 

Easy-to-use reporting and analysis tools built “on top of’ the data repository(ies) 
will help stakeholders quickly access and analyze key data. These tools may 
include statistical and visual analysis tools that help identify trends and 
relationships among variables; and may be consolidated in, or accessible from, 
user dashboards to help educators and other users explore key information in real 
time. 



Capacity to Use and Understand the Data 

If users do not understand what data are available and how to appropriately analyze and 
interpret them, the information will not be used or, worse, may be misused. Therefore 
users must be trained in data analysis and its predicated technology, and must understand 
the value and limitations of the data at their disposal. 

> Effective dissemination 

Access, reports, and tools are not enough. Potential users must know the 
information is available and understand how to use it effectively. Agencies 
should thus communicate what data assets are, or will be, available. 

> Training and professional development 

Stakeholders should be trained to effectively use the available tools to access, 
query, and analyze the information. This professional development, both pre- 
and in-service, should also teach how to appropriately interpret and use the data. 

> Staffing capacity 

Agencies must be able to understand and analyze their data. In-house analysts 
who understand the data and can use them to answer questions and build ad-hoc 
reports can be very beneficial. Alternatively, or additionally, agencies may look 
to outside researchers (e.g. from universities) for analytical support. 
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Support from Leadership and Peers 

All the investment in training and technological tools will be wasted if staff are not given 
the time to use the available resources and apply what they have learned to improve 
educational processes and student outcomes. 

> Leadership support for data use 

Leadership support and a strong culture of data use to improve learning are 
critical to effective data use. Educators need help to successfully decipher the 
data and apply the insight they gain to instmctional planning and practices. 

This help often comes from principals, but may also come from others such as 
department lead teachers, in-house data coaches or hired consultants. 

(Office of Planning, Evaluation, and Policy Development 2009) 

> Organizational support for data use 

Educators should collaborate to learn from their colleagues, consider their 
experiences, share best practices, reflect on the data, and formulate strategies. 

This requires flexibility in the pace of the curriculum and openness to teacher 
innovation. If teachers are to use data to help their students, they must be 
able to adjust their strategies when necessary. If the curricular schedule is 
inflexible, teachers will be unable to respond to the data effectively. Additionally, 
standardized frameworks, procedures, and tools to guide data use can help 
educators leap from raw assessment and other data to instructional changes. Tools 
for this purpose may include data system links to instmctional materials and 
model lesson plans, as well as to instructional frameworks and curriculum guides. 

(Rennie Center for Education Research and Policy 2006) 

> Time 

If educators have access to masses of high-quality data and state of the art 
technology, but are not afforded time to review reports or interpret key data, the 
information will probably be ignored. Education leaders at every level should 
build time for data use into administrators,’ teachers,’ and other staff s regular 
schedules. This may require some creative scheduling and perhaps some extra 
staffing, but it is essential. 



Quality, Access, Understanding, and Support 

Once these four prerequisites of data use are met, the stakeholders must then take the 
final step and use the tools and the new data to inform their work. Identify what works 
and what does not, adapt to meet the challenges the data have revealed, and increase 
operational efficiency. 
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"arly in the Oklahoma morning, the superintendent ot Western 
■ Heights (WH) Public School District sits down at the computer 
and logs onto his personalized dashboard. There, he peruses a host 
ot data on the district's schools. Noticing a new dropout at the high 
school, he drills down to look at his record, checking the former 
^ student's expected graduation date and test scores; and exploring 

demographic, economic, and other data to try to discover what may 
have contributed to the loss. Meanwhile, teachers and staff take 
roll and enter attendance records into the local data system, automatic notifications are sent out, and attendance 
secretaries at each school contact the parents of absentees. Later that day, several teachers meet to discuss results 
of a recent formative assessment and to devise a joint plan to meet their students' needs. They view fresh data 
on their students' performance, seeing which of them has the most success with which areas of the curriculum, 
sharing effective strategies, and using the district's Professional Learning Community (PLC) website and curriculum 
management system to find helpful instructional materials. While on the PLC website, which links to the district's 
many other teacher groups, they post a question to their colleagues. That evening, a state university statistician who 
is also the district's school improvement director visits the site and reads the question. He begins to gather data 
from the LDS to try to find the answer. 

Linder strong leadership from a superinfendenf who placed high value on dafa use long before fhe issue was 
prominenf on fhe nafional or sfafe educafion agenda, Wesfern Heighfs has created a robust local LDS and paired 
it with extensive access, sophisticated analytical capabilities, and a strong culture of data use. The combination is 
a recipe for very effective, data-informed decisionmaking that has gained national attention. 



Western Heights dashboards 

Administrator dashboard: Administrators have access to a host of school-level summary data, including 
the number of suspended studenfs, meal sfafus, recenf enrollmenf and fransfer counts, and dropouts; 
a trend-based analysis tool that shows performance on the state annual summative assessments; and 
an NCLB validation requirement report, that shows the percentages of studenfs tesfed in each school by 
subgroup and by feacher. 

Teacher dashboard: Teachers are given alerts when their students are falling behind or having frouble 
with particular areas of fhe curriculum (e.g., which sfandards they are struggling to master), class 
summary reports on test results by academic standards, reports that show student test scores and 
attendance data, trend data on students' performance by academic sfandards, and more. 



Parent dashboard: Parents can easily access data on their children, including attendance, schedule, 
meal account, report cards, and standards-based reports on student performance. 




System: All the pieces that help make the district a model data user 

In building its district-level LDS, Western Heights decided not to rely on any single vendor. Instead, the district 
pieced together a collection of inferoperable producfs— all Schools Inferoperabilify Framework-complianf 
(http://www.sifinfo.org/us/index.asp), "besf-of-breed" fechnologies. The disfricf's dafa infrasfrucfure is made up 
of a dozen or so separafe buf interoperable systems, including a student information system, a student ID system, 
a data warehouse, food services and fransporfafion sysfems, and ofhers. Each sysfem has clear authorify over 
cerfain dafa ifems and aufomafic updafing fhaf ensures fhe dafa are current and consistent districtwide. 
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Beyond this foundation, the district's system and approach include 

■ Formative assessments: A number of formative assessments are used to gauge student performance 
throughout the school year. 

■ Standards mapping: Tools map a host of state assessments to a common set of academic standards. 

■ Dashboards: Dashboards provide easy access to detailed historical student data, including educational 
inputs (e.g., teachers, schools attended, program participation, etc.) and outcomes (multiple measures 
including assessments mapped to standards), and an array of analysis tools. Administrators, teachers, 
and parents have dashboards with tailored access and functionalities. 

■ Cohort-tracking system: Recent and historical data use identifiers to follow students and cohorts, and 
to establish appropriate graduation expectations. Three levels of cohorts are followed: local, state, and 
federal. This gives the district a fairer way to assess value added and accurately assign responsibility 
for successes or failures. For instance, if a student comes to the district in 1 0th grade with an expected 
graduation year of 2011, but actually graduates in 2010, the district can rightfully take credit for that 
expedited success. However, if the student was held back in an earlier grade while in another state or 
district, the federal and/or state cohort might be 201 1 but the student can be placed with the local 2012 
cohort. If the district does its job, the student will graduate in 201 2 and the district will not be blamed for 
falling short of fhe state- or federally expected graduation year. 



The pros and cons of perceptions data 

Western Heights collects perceptional data— survey data measuring perceptions or "feelings" about the 
learning environment— from teachers, students, and parents to inform and evaluate policy decisions. 
While helpful in many ways, these data can be problematic. 

In the first place, they are difficult to collect. Due to limited computer access in the community, 
electronic surveys were impractical. The district therefore used paper surveys during enrollment and 
parent-teacher meetings, which led to data-entry problems. Frequent surveying lowered response rates, 
so the district decided to conduct surveys only when they had a clearly defined purpose. For example, a 
survey was conducted to gauge opinions about the restructuring of academic scheduling. Surveys were 
also used in some cases to try to understand why student performance had dropped. 



In addition, the district found the data limited in terms of application. Overall, staff found that while 
perceptions data were useful for informing decisions in the present, they were not well-suited for 
longitudinal applications. Subjective changes in opinions or perceptions, the district realized, do not 
represent change in the way that objective data such as assessment scores do and, thus, do not track 
well overtime. 




■ Family information tracking: The district assigns family IDs during enrollment rather than assigning 
a different ID tor each child. These are used for various purposes, such as allowing parents to access 
information on all their children districtwide, with a single user ID and password. Family IDs can also help 
ensure all students eligible for free and reduced-price lunch receive program benefits. 

■ Growth model: A sophisticated and publicly accessible growth model is being developed to measure and 
display growth and value added. The model relies on the district's common classification of academic 
standards. 



Book Four of Four: Advanced IDS Usage 



21 





■ Automatic reporting: The system allows the district to produce automated state and federal reports in 
minutes. 

■ Professionai iearning community (PLC) network: A districtwide community of teacher PLCs collaborates 
to improve instruction and student achievement, supported by a website that facilitates communication 
across the district. 

■ Research support: A university researcher works with the district, providing data analysis and 
instructional improvement support. 

■ Merit pay: A system for creating merit pay is being developed, based on a host of teacher and student 
data. This evolving project has been widely supported by stakeholders, including teachers and the 
teachers union, who view it as a fair approach. 

What is the district doing with its LDS? 

Digging deep with outside assistance: To better understand the roots of its challenges and improve its students' 
outcomes, the district brought in an outside statistical consultant to analyze its data. Administrators wanted to 
build predictive models that could use scores from two annual norm-referenced tests to forecast how students 
would perform on the state's high-stakes, year-end test. The consultant used the district's norm-referenced 
and state criterion test data from a previous school year to create models for each grade level and subject. After 
demonstrating the model's strong predictive value, he applied the formulas to the current school year and created 
reports that showed teachers where their students were struggling. This allowed educators to focus on problem 
areas and better prepare students for the year-end assessment. Similar models are now being developed with 
more frequent benchmark achievement data; this will help the district predict end-of-year assessment performance 
as well as show academic growth. Additionally, these models will help assess the benchmark tests themselves to 
determine if they are rigorous enough, and if instructional practices are adequately preparing students. 

Beyond the predictive models, the consultant worked closely with the district to observe how staff used the data 
and software applications. He then analyzed and presented findings, worked with teacher groups, and even held 
one-on-one training sessions to help educators determine what to do with the available data. This new insight 
encouraged teachers to work more openly and collaborate more effectively to meet student needs. 

Teacher collaboration: Western Heights teachers work together in professional learning communities (PLCs), 
teaming up to analyze data and devise strategic improvement plans. Some teachers are trained as "instructional 
leads," learning to work with teachers and data to determine which strategies are working and which are not; and 
in which content areas students are struggling, both individually and as a group. They also use the data to Identify 
which teachers have the most success with different students and areas of the curriculum. Students are then 
placed with teachers who possess particular strengths suited to their needs (see discussion below). Teachers also 
take advantage of the district's curriculum management system to find lesson plans and resources, and to send 
students materials to help them as necessary. 

Collaboration does not end with individual PLCs. Teachers work together not only within subjects and grades, 
but also across grades to make sure students are ready for the following year. Teacher groups are linked by a 
central PLC website that allows PLC members across the district to communicate across grade levels; and the 
entire school district to ask questions, provide answers, and share instructional resources. The district's statistical 
consultant also actively engages with the staff through this communication tool. 
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An example of data analysis to inform policymaking 

Leaders of Oklahoma's Western Heights School District were determined to identify the factors related 
to poor student performance in their schools. They hired a research consultant to collect a range of 
student data and conduct a series of analyses examining the relationships between test performance 
and variables such as socioeconomic status (SES) and attendance. The consultant found that, contrary 
to expectations, SES had no strong relationship with student performance. Attendance, however, was 
strongly correlated with test scores. Based on these findings, the district published a report suggesting 
that students with less than a 90 percent attendance rate were more likely to do poorly on the state's 
year-end test. Though no causal relationship was proven, the findings prompted the district to take action. 

Policies were implemented to increase the number of students who showed up to school. For example, 
an attendance secretary was added to each school and an attendance coordinator to each professional 
learning community (PLC). Parents of absentees received automatically generated notifications and staff 
worked to maximize the number of students who showed up to school. And the district's superintendent 
even asked that the district start enforcing its school attendance laws more aggressively. 



Further analysis unearthed a more fundamental issue: attendance problems were highly correlated 
with mobility. In this district with a highly mobile student body — a 70 percent mobility rate — the most 
transient students, regardless of SES, were most likely to have the lowest attendance rates and the 
lowest assessment scores. Faced with this reality, the district is attempting to identify other factors that 
may contribute to the relationship between mobility and student success. 




Matching teacher strengths to student needs: By identifying detailed information about student needs (the areas 
of the curriculum in which they are struggling as identified by online, standards-based testing) and teacher 
strengths (the areas of the curriculum in which they have produced the greatest student performance gains), 
the district moves students to different teachers throughout the school year to maximize learning. This is done 
in various ways to provide the best instruction available as evidenced by the data. For example, if a teacher has 
notable strength teaching a particular concept, others may bring their students to that teacher to learn that portion 
of the curriculum while they themselves focus on other content areas in which they have the most success. In 
another example, students having trouble with a particular skill, perhaps spelling or long division, might be placed 
with a teacher best equipped to teach this skill. Teacher-student pairing is also used in certain interventions such 
as after-school tutoring. This strategy has been very effective in helping students gain the skills they need to stay 
on track to success. 

The future 

It took Western Heights years to establish its sophisticated system, and to change its culture to not only use but 
rely on data for decisionmaking. Today, the district continues to expand its data use capabilities; and to delve 
further Into its data to better understand its challenges and find effective solutions. 
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Chapter 3 



WHAT DO Stakeholders Get? 
LDS DATA USES AND USERS 



he benefits agency stakeholders will reap from the LDS depend on the effort 
invested in planning, designing, developing, populating, and managing 
a system that will help improve student outcomes. In the end, education agencies 
have to deliver on their promises and deliver an LDS that has practical value for users. 
Ideally, access to high-value data— with appropriate privacy protection— will be extended 
to state and federal program administrators, education researchers, district and school 
leaders, teachers, parents, students, the general public, and organizations outside of K-12 
education. 



Already, some administrators are using the data to more accurately and efficiently 
administer or comply with federal and state programs and requirements. Some are casually 
using single points of data to check the status of schools or students by viewing end-of- 
year summative assessments, periodic formative assessments, schedules, attendance and 
discipline records, etc. Some are producing reports or using real-time dashboards to get 
frequent updates on key information and trends; and turning the information into changes 
in policy, curriculum, training, and classroom instmction to improve education outcomes. 
And some are conducting sophisticated analyses by combining detailed data integrated 
over time and across institutions, matching students to teachers and programs, measuring 
student growth, and gauging the value of various educational inputs. 



Don't leave data use to IT 

Figuring out how to use the data should not be IT staff's responsibility. 
Ask business staff and researchers. 




This chapter provides an overview of how some stakeholder groups might use an 
LDS equipped with key data and helpful tools. Building on earlier sections of this guide 
series, it reviews many of the benefits an LDS can provide. 
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District and School Leaders 

Superintendents and other district administrators, principals, directors and other school 
staff, school board members, and other local-level leaders are often more avid data users 
than other stakeholders. This group can benefit greatly from access to real-time, detailed, 
longitudinal data. These users may, for instance, rely heavily on both aggregate and 
individual student- and staff-level data to guide decisionmaking about financial and human 
resource allocation, hiring, professional development, staff promotions and compensation, 
facilities maintenance, disciplinary policies, student remediation, placement, promotion, 
program implementation, product purchases, and much more. These local leaders may use 
portals, dashboards, or other business intelligence tools to view key indicators and trends; 
evaluate schools, principals, teachers; and assess the impact of programs and policies on 
long-term success and student learning between marking periods or from year to year. 
Depending on the flexibility of the tools available to them, administrators may be able to 
highlight information for themselves and their staff by producing their own reports, charts, 
graphs, lists and more. Using techniques like the value-added assessment models discussed 
later in this chapter, administrators can more readily identify the most effective teachers; 
match teachers with the training they need to hone their skills; promote the most effective 
educators to lead positions; facilitate the sharing of best practices among staff; inform 
compensation decisions; and determine which certifications, trainings, and other programs 
are worth the investment. 



Data use: Culture change! 

D ata use, or lack of use, is a cultural issue. Education agencies and schools 
must shift from reporfing and collecting data for compliance to facilitating 
data use for educafion improvemenf. Leaders should ask: 

■ Whaf value is puf on dafa use? Are dafa used only for operafional and 
compliance reasons, or are fhey also used to inform decisionmaking? 
■ Whaf is envisioned for data use? Is there a clear plan to achieve this 
vision? 

■ How is that vision implemented by leadership in individual schools? 

■ What emphasis is put on data quality? 

■ Is support available to help staff use dafa? 

■ Do educators have time allotted specifically for dafa use and 
reflection? 

■ Are the tools for accessing and analyzing dafa user-friendly? 

■ Whaf is fhe depfh of available analyses— progress monitoring, 
benchmarking, prescriptive, etc.? 

■ How is the information the data reveal applied? 

■ How frequently are data collected about student learning, the 
educational environment, stakeholder perceptions, etc.? 
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H “(At) the moment, high school principals know virtually nothing 
about what becomes of their graduates. Most don’t even know 
whether their students make it to college at all. 

What data th^ have is anecdotal. ‘Once a graduate happened to drop 
by and tell us she was struggling with college writing, ’ Linda Calvo, the 
principal of Arleta High School in Los Angeles, told us. ‘We changed our 
writing curriculum based on what she said. But her visit was a 
totally random occurrence.’ ” 

(Schramm and Zalesne 2009) 




The following are some specific questions users might answer with LDS data: 

> How well are a school or district’s former students faring in later grades, 
postsecondary institutions, or the workforce? Based on the data, administrators 
can devise evidence-based plans for improving student outcomes. 

> What is the agency doing well and what must be improved? For instance, are 
many graduates requiring college-level remediation in a particular subject? 

> What should improvement goals be? With goals set, educators can use the data to 
monitor progress towards those targets. 

> Which schools and teachers might serve as models for those having less success 
preparing students? 

> Do certain teachers have more success with students of similar socioeconomic 
status, race/ethnicity, sex, prior performance level, or other characteristic? 

> How well do local curricula, materials, and assessments align with state summative 
tests? How can they be better aligned to ensure students succeed? 

> What was the impact of any new textbooks or other instructional resources? 

> How are certain student characteristics affecting the school or district’s overall 
performance on assessments (English language learners, transfers, etc.)? 

> What are the local rates and trends in attendance and mobility? How do these 
factors impact student performance? 

> Who is dropping out of the school or district? What signals should prompt staff 
to intervene with struggling students and prevent future failures? 

> Are school or district educators equitably distributed in terms of qualifications? 
Are certain geographic areas, subjects, or grades disproportionately serviced by 
the least effective or least qualified teachers? Are all district schools adequately 
staffed? 

> What skills do teachers need to strengthen? Based on past outcomes, which types 
of professional development produce the greatest results? 

> What is the value added by individual teachers, teachers with certain characteristics 
(training, certifications, etc.), grade- or subject-specific instructional strategies, or 
programs? Do these effects vary by student characteristics or other factors? 
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> Which programs are working and which are not? What weaknesses within 
otherwise successful programs might be improved? Does a specific program have 
a stronger impact on students with certain characteristics than on others? 

> What facilities maintenance is needed?"' 

> How well do students in a specific school (or schools) for a number of years 
perform in comparison with children enrolled the same amount of time in the 
highest-performing comparable school(s)? (Dougherty 2002) 

> Are other comparable schools having greater success in particular subjects or 
content areas, or with students sharing certain characteristics? If so, what can be 
learned from them? 



Focusing the lens with better data 

D etailed, longitudinal data can provide many types of insights. Knowing that a school's 
3rd grade achievement gap has grown since the prior year is useful. But it is much 
more constructive to be able to account for changes in the school's student population, 
and know whether the achievement gap in this year's 3rd grade cohort of students actually 
closed relative to those same students' scores during the prior year. 

Similarly, a teacher should know that the Latino students in her school lagged White 
students by 1 0 percentage points on this year's state math assessment. But it would be 
even more useful for her to know whether a particular Latino student's math scores were 
proficient before the student entered her class. And it would be even more valuable for 
her to know the student had difficulty in fractions and, specifically, that on the last interim 
assessment, the student repeatedly made the common mistake of adding the denominators 
together. Based on this information, that teacher can work with the student on this skill, or 
seek help from other teachers who have been the most success teaching this skill. 



Teachers 

Teachers could potentially reap significant benefits from access to detailed, longitudinal 
student data. These users on the front lines of the education system can use the data 
to monitor individual performance on assignments and assessments, as well as growth 
through the year; identify struggling students as quickly as possible; target extra help to 
students having difficulties; adjust lesson plans as needed; and set class-level improvement 
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Administrators can use LDSs to identify maintenance needs, compare the physical conditions and maintenance needs 
of buildings and utility systems across a district, and plan and take preventive action to ensure long-term facilities 
sustainability. 
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goals. To ensure all students’ needs are met, instruction may be tailored for small groups 
or even individual students in a class. Teachers may access data in the form of reports 
produced by the state or district to provide key information on student- and classroom- 
level performance. Or, they might use more high-tech solutions like analysis tools or 
personalized dashboards that present key, real-time information that can be drilled down 
to the student level. Teachers can also use the data to identify their own success and 
failures in the classroom, share best practices with colleagues, and seek better solutions 
based on the facts. Some specific questions and might be: 

> In what areas of the curriculum are students (the entire class, subgroups, or 
individuals) excelling or stmggling? 

> Are students on target to meet proficiency or academic growth targets? Are any 
students showing signs that they may be heading in the wrong direction? At 
what point did these students start having trouble in a subject or area of the 
curriculum? 

> Are students with particular characteristics doing better than others? What might 
this say about teaching style or other factors? 

> Are other teachers having greater success teaching particular subjects or academic 
standards? If so, what can be learned from them? 




Parents and students 

Though the scope of an education agency’s LDS accessibility may not reach students and 
their parents, access to a student’s course schedules, attendance, disciplinary record, credits 
earned, summative and formative assessment scores, program enrollment, and other similar 
data can be very useful. Such access can serve many purposes, including setting goals, 
directing study, and preparing for meetings with teachers. These stakeholders can use the 
data to answer the following types of questions: 

> How is a student doing on assessments this year? Is the child on track to meeting 
the growth target or the proficiency standard? 

> In what areas of the curriculum is more study or extra help required? 

> What courses should the student take next year? 

> Based on grades and test scores, what colleges would be appropriate? 

> What grades need improvement to get into the first-choice college? 

> What programs might benefit the student? 

> How does the school compare to similar institutions in the district or state? 



Researchers 

Longitudinal data sets contain all the students across a school, district, or state, and many 
years of data for each, providing a vast source of detailed information. This should allow 
less dependence on aggregated data, surveys or studies that rely on samples of students 
that may or may not be representative of the larger population. Using more complete data 
will, in turn, allow researchers to derive results that are more reliable and representative 
of the broader education system. The availability of data should contribute greatly to our 
understanding of the education system (e.g., to evaluate the effects of programs, policies, 
curricula, teaching practices, as well as external factors and more). State education agencies 
may grant researchers access to data through various means, and in accordance with 
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relevant state and federal privacy laws. These may differ, as states vary in their stance on 
providing personally identifiable data to researchers: some allow it under strict guidelines, 
while others require de-identification of all data before sharing with authorized researchers. 



Self-service 

research 



f 

While some states 
have trouble keeping 
up with data requests 
trom researchers, others 
have solved this issue 
by creating a "selt- 
service" model. These 
states grant approved 
researchers access to 
a central data store 
containing pre-packaged 
data sets. 



Research partnerships, data-sharing agreements 
and memoranda of understanding 

W hen an education agency has collected potentially useful data, but lacks the in- 
house staff necessary to analyze them, it may enter into an agreement with outside 
researchers to get its questions answered. The agreement may take the form of a research 
partnership, a data-sharing agreement, or a memorandum of understanding (MOU); the 
researchers may be independent research organizations, postsecondary research centers, 
the National Center for Analysis of Longitudinal Data for Education Research (http://www. 
caldercenter.org), etc. This can be a mutually beneficial arrangement. For instance, the 
Florida Department of Education provides authorized researchers with a list of subjects 
they want studied. In return for access to the data, the researchers conduct analyses to try 
to answer the provided questions and, at the same time, may also be allowed to explore 
their own questions, perhaps if fhey agree to share findings with the agency before they are 
published (Dougherty 2008). In addition to facilitating potentially enlightening research 
at minimal cost, education agencies may also benefit in other ways, such as having the 
data cleaned and converted into more useful formats by the researchers (National Research 
Council 2008). 

Some states require this type of agreement before access to data is granted. An MOD can 
be established between an education agency and a research organization or other party 
such as a postsecondary institution, another agency within the state, or a state education 
agency from a different state. This allows data sharing or access to protected data. MOUs 
generally articulate the agency's interpretation of the Family Educational Rights and Privacy 
Act (FERPA) and other relevant privacy laws, establish ownership of the data, set the terms 
of data use, and establish a common set of goals for the data analyses. 



State-level decisionmakers 

Governors, chief state school officers, and legislators are some of the leaders at the state 
level who can use LDS data to learn about their state’s student population, including 
program eligibility, mobility, etc. They might also monitor student performance and 
outcomes, such as academic performance, dropout rate. National Governors Association 
(NGA) or four-year-cohort graduation rate,"' etc.; apply more sophisticated methods of 



"The National Governors Association (NGA) graduation rate is a standard, four-year adjusted cohort graduation rate 
agreed upon by all 50 governors in 2005. 
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accountability, including value-added and growth models; and inform decisionmaking 
on policies and programs, resource allocation, and initiatives to help improve their state’s 
education system. (DQC 2009) 



Sample state data use websites 

School Improvement in Maryland: http://www.mdkl2.org 

Wisconsin Information Network for Successful Schools (WINSS): http://www.dpi.s1ate.wi.us/sig 
Indiana Accounfabilify System for Academic Progress (ASAP): http://www.doe.state.in.us/asap 



Federal government and state education agencies 

A successful LDS should enable more efficient, less burdensome data reporting by schools 
and districts; and help state education agencies (SEA) meet federal reporting requirements 
such as timely and accurate submission through the EDFacts system. State-level 
decisionmakers may use these data to inform decisionmaking about funding and other 
issues, including the improvement of state-funded teacher preparation and certification 
programs, dual enrollment, charter school policies, and more. (DQC 2009 and National 
Research Council 2009) 



Planning big, starting small 

A llhough educafion agencies may have high expeclalions aboul revolufionizing 
dala use, Ihey likely will have lo achieve Iheir goals on a small budgel. As such, 
Ihey should idenlify Ihe mosl crilical issues for stakeholders and address these first. 
Achieving critical goals first helps garner support from skeptical stakeholders. 



Higher education, including teacher preparation programs 

Postsecondary leaders can use LDS data linked to K-12 to prepare for their incoming 
students’ needs, and to better assess applicants’ credentials and the predictive value of 
their admissions criteria on postsecondary success. They can also use the information 
to ensure the necessary staffing resources and to identify the most effective transition 
strategies (DQC 2009). Furthermore, if teachers’ credentialing programs are linked to 
student data, those programs can better assess their own effectiveness by examining the 
educational outcomes of their graduates’ students. Additionally, these institutions and 
programs can use detailed LDS data to understand the geographic distribution of critical 
shortages schools face in hiring teachers of mathematics, science, English for speakers of 
other languages, special education, etc. They can then use this insight to respond more 
strategically to this problem, advising students into these subjects and offering field 
experience in schools facing the direst shortages. 



Book Four of Four: Advanced LDS Usage 



31 




Industry, workforce agencies, and the workforce development 
system 

Industry can use longitudinal student data to learn more about tomorrow’s workers, 
perhaps considering a state’s educational outcomes when deciding where to establish 
businesses (EIMAC 2007). Workforce development programs can also assess their impact 
on graduates’ success in the labor market and identify areas that need improvement. 




Early childhood education 

Longitudinal data linked from K-12 to early childhood education (ECE) programs can 
show administrators how well they are preparing children for success in elementary school. 
In turn, the results can help them identify ways to improve their services so that students 
get the best start possible. (DQC 2009) 

Social service agencies 

Foster care, juvenile justice, health and human services, and other social service agencies 
that share data with education agencies can use the information to study how their 
services affect students’ educational outcomes (DQC 2009). For example, the connection 
of education and foster care data can help agencies better understand the needs of their 
clients; the relationship between their services and academic performance, such as the 
impact of student mobility across foster homes and schools on academic performance; and 
the ways they can improve their services to better meet children’s needs (Rennie Center for 
Education Research and Policy 2006). 



The public 

The public in general also stands to gain from more detailed and accurate data. Taxpayers 
can understand with greater precision where their money has gone, and how well students 
are performing as a result of the resource allocation decisions made by school and district 
officials. Homebuyers will also have better access to information on school quality in the 
community, a factor in property values and on the education their children receive. 



Balanced Assessment System 

Assessment results are essential to any EDS focused on improving the effectiveness of 
educational processes and supporting increased student learning. However, what constitutes 
high-quality, useful student assessment data is changing. For decades, educators have relied 
primarily on large-scale, summative assessments to judge student performance and to hold 
schools accountable. An increasingly large body of research suggests that, while these 
assessments estimate achievement for groups of students at a specific time, they cannot 
inform instmctional planning for the individual student. 

More importantly, these studies suggest that, for a significant and positive impact 
on learning, a well-designed student assessment system should balance assessment types 
and purposes to include periodic summative assessment with frequent teacher-generated 
formative assessment, and ongoing self-assessment by students. This more balanced 
approach provides teachers with useful information for adjusting instruction to improve 
student learning, and helps students better prepare for future assessments. Furthermore, 
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the integration of balanced assessment data with other kinds of LDS data can broaden the 
scope of questions that can be answered. For instance, stakeholders gain greater ability to 
assess the effect of instructional strategies and programs, return on investment, and the 
relationships between policy and learning. 



Formative assessments 

Formative assessments are relatively short tests that are given 
periodically throughout the school year "to monitor student progress 
during instruction." 

Source: Rennie Center for Education Research and Policy (2006) 

Formative assessments are relatively short tests that are given frequently throughout the 
school year “to monitor student progress during instruction” (Rennie Center for Education 
Research and Policy 2006). These tests can be a powerful component of a balanced 
assessment system. While summative assessment data derived from annual tests may 
be sufficient for high-level decisionmakers, local-level educators and administrators will 
benefit from the more frequent formative assessment data to help monitor student learning 
throughout the school year, guide curricular and instmctional planning, differentiate 
instruction for individuals and groups, and target interventions. 

However, education agencies must think carefully about their specific needs when 
they purchase or build a formative assessment system. For more information and tips on 
formative assessment solutions, see Data-Driven Teaching: Tools and Trends (Rennie Center 
for Education Research and Policy 2006). 




Measuring Progress in Different Ways: Growth 
Measures, Growth Models, Value Added 



Two common uses of longitudinal student- and staff-level data are growth models and 
value-added models. While both models focus on student academic growth, they approach 
this growth from different directions. Growth models consider past academic performance 
to look forward and set goals for future student growth; value-added models look back on 
progress that has been made in order to assess the effectiveness of particular educational 
inputs. Growth measures are the assessment tools used to feed these models with data 
about student growth, such as formative or interim assessments. Thus, growth measures 
feed value-added and growth models that calculate results in order to tell us about the past 
and help us plan for the future, respectively. 




Growth measures 
Growth models ^ 
Value-added models 




Growth measures are frequent, short diagnostic assessment tools 
(e.g., formative and interim assessments) used to measure growth in 
academic performance. 
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Growth models 




Growth models ore statistical models that measure academic growth 
over time to predict future performance and assign target goals that, 
if met, will bring students towards a proficiency standard over a 
period of time. 



Growth models measure academic growth over time. This measured rate is commonly used 
to establish performance benchmarks for points in the future (e.g., grade levels) against 
which to gauge student progress over time. 

This so called “growth to proficiency” method has been supported as a fairer 
approach to measuring performance and holding schools accountable than snapshot 
assessment scores and uniform proficiency thresholds that do not take into account prior 
performance and other important student characteristics (Barone 2009). Growth models 
are also respected as valuable tools to help teachers plan instruction, target students for 
intervention, and help them meet their performance goals. Proponents also have argued 
that this approach encourages educators to help all students improve, instead of focusing 
disproportionately on children on the cusp of proficiency at the expense of those far below 
or above the proficiency level in order to seek adequate yearly progress (AYP). Commonly 
cited limitations of these models include a lack of transparency about how results are 
calculated, and a narrow scope that may only focus on assessment results as measures of 
academic performance. 



Value-added models 




Value-added models are statistical models that use a range of data 
on student academic growth and other characteristics to estimate the 
impact of certain educational inputs. 



Value-added models analyze student assessments over time to estimate the relative effects 
of various educational inputs (schools, programs, curricula, teachers, etc.) on student 
learning (Wilson and Yeagley 2007). These models are commonly applied at the individual 
student or classroom levels. 

Value-added models may be used for a wide range of purposes, such as: 

> to identify the most effective teachers and help less successful ones improve their 
craft through collaboration, professional development, or other means; 

> to measure the effects of programs, curricula, or instructional strategies; 

> to evaluate the effects of principals and schools on student performance; 

> to group students and teachers based on teacher strengths and student needs or 
characteristics such as disabilities, English language learner, etc.; 
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> to assess the distribution of teaching talent and guide redistribution or 
recruitment efforts; 

> to inform teacher compensation and promotion decisions; 

> to identify the most successful schools and hold the least effective ones 
accountable; and 

> to identify the most and least effective teacher preparation programs, and ensure 
that all pre-service training programs are producing effective teachers. 



Measuring the value added by teacher preparation programs 




A small number of states are using value-added models to evaluate teacher 
preparation programs. By measuring student growth through such a model, 
linking those student data to teachers, and tracking back to the institutions 
where the educators received their pre-service training, states and districts 
can identify the most effective teacher preparation programs. They can also 
work to improve and hold accountable the programs that produce the least 
effective teachers. 



A prerequisite to value-added assessment modeling is the linkage of individual 
student- and teacher-level data. This issue alone has been a considerable source of 
contention; however, the application of value-added data as a basis for accountability has 
added another layer to the debate. 

Critics have raised concerns about these models’ ability to prove causality (they 
cannot), and they question whether it is therefore appropriate to base teacher evaluations and 
compensation decisions on these data. Others point out that the models are limited in scope 
and do not present very broad views of student performance, putting too much emphasis on 
assessment results and, even then, often only assessments in core academic subjects. 

On the other hand, proponents argue that value-added models offer a fairer 
approach to evaluating teacher impact than other common indicators of student 
performance and proficiency levels. They favor value-added assessments because these 
consider a range of growth-measure data in gauging academic performance. These models 
can also factor in variables over which teachers and schools have no control, such as 
students’ socioeconomic status or prior achievement in other classes or schools. 

Not without their problems, value-added models nevertheless represent a promising 
application of longitudinal data. Already, many different models are being used by states 
and districts across the country, and practitioners are working to refine them in order to get 
the most accurate and reliable results. 
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be part 
vendor 



A head of the pack in many aspects of IDS development, the 
, Arkansas Department of Education was struggling to make 
its data useful fo sfakeholders. A researcher working with the 
state had pointed out that simply giving teachers and other users 
access to raw data was not enough; not enough to facilitate 
instructional improvement, and not enough for program evaluators 
^ to easily analyze the data for fhemselves. Convinced if needed a 

beffer approach, Arkansas decided fhaf a visual applicafion mighf 
of fhe solufion. So, in addifion fo developing a dashboard sysfem and scorecards, the agency worked with a 
to develop a visual analysis tool to help users examine the state's data. 



After about a year of development, the state rolled out "Hive" (dubbed after grid computing, not bees), which is now 
available online to the general public. An adjunct site gives users quick overviews of their data. The tool is loaded 
with state data, but can also be populated with local level, test, and demographic data by district administrators. 
System designers hoped to empower users with this sleek and simple web interface fhaf allows stakeholders to 
explore test scores at the district, school, classroom, and individual levels.* Filter options for program participafion, 
demographics, and socioeconomic sfafus allow users fo view additional details or specific records of interesf. 

In addition to displaying assessment scale scores. Hive incorporates the Colorado Growth Model and allows users 
to place student growth percentiles on one axis and scale scores on the other.This adds another dimension, 
with scatter and bubble plots showing not only how a school, class, or student did on an assessment, but also 
improvement since prior tests. The tool can display scores or growth in two subject areas at once. Scatter plots 
show each of fhe individual unifs of analyses (e.g., schools or sfudents) and users can roll fheir mouse over 
individual poinfs. In fhe case of sfudenfs, fhe individual's sex, race, efhnicity, and socioeconomically disadvanfaged 
sfafus are displayed, along with names if fhe user is aufhorized to see them. Hive's ability to aggregate students' 
scores and growth by teacher has allowed authorized users to explore the patterns in school- and classroom-level 
performance. This allows leaders fo learn from successes. In some cases, schools have been idenfified in which 
every teacher's students are performing and growing af excepfional levels.The stafe is using fhis informafion fo 
examine fhe underlying factors that have contributed to exceptional student performance. 

Hive lets individuals work on their particular area of interesf: a parenf interested in comparing schools in their 
district, a teacher trying to discover patterns of performance for his studenfs, a principal frying to determine if 
fhere are weak areas of insfrucfion in her building, efc. However, while Hive provides individual users wifh many 
analytical and visualization capabilities, the tool's website was also designed as a venue for collaborafion as if 
allows the exchange of findings and ideas. Users can posf and share graphs they created, comment on the work 
of ofhers, and imparf ideas in fhread discussions. This allows users across fhe sfate fo learn from colleagues and 
ofhers wifh similar inferesfs.As a result, knowledge about student achievement in the state as a whole is greatly 
increased. 



To prepare Hive users to explore and analyze the state's data, a training course has been offered af regional 
service cenfers. School and disfricf staff parficipate on a volunfary basis; while adminisfrators are required fo take 



'^Initially, no classroom-level aggregation was included; this was added when teachers showed great interest in being able to view 
their impact on student performance. Careful attention has been paid to protecting the privacy of individual students. For instance, 
only authorized users are given access to identifiable information. Pages accessible to the public censor student names and suppress 
sufficiently unique demographic information that could be used to identify individuals. 
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three hours of data analysis training, and teacher certification requires technology training that may be comprised 
of data analysis. Rather than taking an abstract approach, the training walks the users step by step through the 
process of requesting and displaying the data to answer specific questions. For example, to see how a school's 
students compare to others in the state in a particular subject area, a trainee might be led through the process 
of finding and graphing scores and growth. To complement the training. Hive has links to helpful resources and 
training materials, including short videos to help users at all levels. A certified administrator from each district 
is also trained to upload local data and extend access to individually identifiable data to teachers and other 
authorized staff. The state uses surveys to gauge the usefulness of the training and the value of the tool in general. 

Arkansas' visualization tool and training program are still young and expected to evolve quickly following user 
feedback. However, the initiative represents a promising, novel approach to education data display and analysis. 

For more information, visit: 

■ The Hive website at http://hive.arkansas.gov 

■ The Hive Quick Looks website at http://www.mrgh.com/ade 

■ The Colorado Growth Model website at https://cdeapps.cde.state.co.us/growth_model_public 
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Chapter 4 




Training for Effective use oe LDS data 




imply rolling out an LDS will not help an agency and its stakeholders 
realize its ultimate benefits. To maximize the return on the system’s 
investment, education agencies should systematically prepare users to take full advantage 
of the system, offering training to help them effectively and appropriately use the 
technological tools at their disposal. Users should also be offered professional development 
(PD) so they know what data are available; develop an appreciation for the value and 
utility of those data; and expand their skills in areas such as data location, inquiry, analysis, 
interpretation, application, and communication. 

District and school administrators, for instance, should be trained to use data for 
evaluation and decisionmaking. Teachers should understand how to use data to guide 
instructional improvement. Legislators and their aides should know how to interpret the 
data and use them to inform policymaking. The press should know how to interpret the 
data to accurately depict the education system. And parents and students should know 
how to track progress and set goals for academic improvement and success. 





Professional Development and Training Issues to Consider 

Professional development and training topics covered may include logging into the system, 
understanding report contents, and using analysis tools for various purposes. Federal and 
state mles, regulations, and policies on data use, privacy, and security may also be included; 
at the very least, informational materials should be made available. Such training will help 
the enterprise hit the ground mnning when the LDS and data become available. 



Need 

Assessing trainees’ needs is important. Before you begin a PD initiative, and iteratively 
along the way, determine what skills your target trainees currently have and what areas 
need to be developed. 
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Current skills and practices 

> What type of data do staff and others already use and what do they do with them? 

> Do they use state assessment data? Local assessments? Other student data? 

> Do they link student data to teacher and other classroom input data? 

> How do they analyze and reflect on these data, and how do they use them to 
improve students’ educational outcomes? 

Current training programs for teachers 

> Do pre-service programs include data use training? 

> What data use training are aspiring teachers receiving at postsecondary institutions 
and through other teacher-preparation programs? 

> Does in-service PD on data use already exist? 



Focus 



Users should know what to expect. This can build excitement and support. Develop a 
focused plan for the topics that will be covered by training and PD. 

Data awareness 

> Will training build awareness of the available data, reports, and technology? What 
new types of data and indicators are, or will be, available through the LDS? 

> What reports will be created? 

> What ad-hoc querying and analysis tools will users have? 

Data use 

> Will the program train users to access, create, and use reports? 

> Will it teach how to access and utilize analytical tools? 




Data interpretation and application 

Beyond pulling data up on a computer screen or even conducting analyses, educators 
should be prepared to understand and act on the numbers. Once they have the facts, they 
must know how to adjust their practices to address problems and improve performance. 

> Will training teach users how to interpret the new data in the reports or the results 
of their own analyses? 

> How will educators and administrators turn raw data, tables, or graphs into 
information that can inform improvement planning or program evaluation? 

Data communication 

> How will you ensure users speak the same data language? If stakeholders are 
going to interpret data and use them to communicate, establishing a common 
understanding of data and the terms used to discuss them is important. For 
instance, if staff have different ideas about what “summative” and “formative” 
assessments are, a certain level of confusion will hinder discussions about the data. 
To get everyone on the same page and avoid confusion, reviewing key vocabulary 
terms or creating a glossary for reference is helpful. When discussing individual 
data elements, a metadata system or data dictionary can also be very helpful. 
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The heart of "data-driven decisionmaking" is decisionmaking 

W hile proponents of data use in education have long emphasized the use of data in 
decisionmaking, a caveat should be kept in mind: people, not data, should ultimately 
drive decisions. In other words, data should be used as a tool to inform decisions, not 
dictate them. While data are invaluable to objective, evidence-based decisions, they can 
also lead users astray if not accompanied by critical thinking. 

Logic and good sense should inform all educational decisions, regardless of what the 
data say. Even if the research says a particular program is highly effective, for example, 
decisionmakers should not automatically implement it. Stepping back to take a look at the 
data or research results is important. Are the data valid for the requested purpose? Was the 
research conducted soundly and with accurate data? Can the results be generalized to the 
education environment? For instance, if the analysis was not conducted with local data, is 
the local student population similar to the sample studied? Can conditions be replicated? 
Does the agency have the financial and staffing resources to effectively implement the 
program? 



Target audience 

> Will PD and training be offered to teachers, administrators, administrative staff, 
technical support staff, researchers, legislative aides, students and parents, the 
public? 

> Will it be mandatory or voluntary for educators or others granted access to the 
data? Will data-use training be tied to educator and administrator certification or 
other form of recognition? 



Provider 

> Who will provide training and PD? For instance, will programs be taught by in- 
house staff with expert knowledge of the system, consultants, researchers from a 
university, or vendors? 

> Will vendors be asked to provide training on the use of their products, both to 
in-house staff managing the system and to end-users? 

> From what level will PD be provided: from the state agency to districts, from 
regional agencies to districts or schools, from districts to schools, within schools 
by data coaches or other skilled staff, etc.? 
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The Forum has more... 

... information about professionai deveiopment and staff training; 

• Forum Guide to Decision Support Systems: A Resource for Educators (2006) 
http://nces. ed. gov/forum/pub_2006807. asp 

(See pages 1 7-1 8 of this Forum document.) 

• Forum Unified Education Technology Suite 
http://nces. ed. gov/forum/pub_ tech_suite. asp 




Format 

> In what format(s) will PD be provided? Will instructors lead face-to-face courses 
with lectures and activities, webinars, pre-recorded self-paced online video 
tutorials, online repositories of training materials, educator collaboration, etc.? 

> How structured will the training be? Will users be taught to follow particular 
steps or procedures when using data, or will training be more abstract? One state 
education agency, for example, taught trainees a structured, step-by-step approach 
to using the data visualization tool. Many felt that this was more useful than a less 
tangible approach. In one basic, yet effective step-by-step data use methodology, 
users first formulate questions (“question posing”); then find, understand, and 
interpret data to help answer the questions (“data location comprehension and 
interpretation”); and, finally, develop an action plan to respond to the reality 
suggested by the data (“data use”). (Office of Planning, Evaluation, and Policy 
Development 2009) 

Scope 

> On what data, reports, and analytical capabilities will the training focus? 
Professional development may focus on a specific type of data if time is limited, 
such as state assessments, but if possible educators should learn how to use local 
assessment and other data as well. Furthermore, workshops that only explain 
assessment data have been found to be less useful than those that teach how to 
translate data into actionable information for instmctional planning (Marsh, Pane, 
and Hamilton 2006). Depending on the range of data, the number of reports and 
tools, and the resources available, an agency may have to decide on the range and 
depth of the training necessary. 
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Timing 

> When will PD be offered relative to system rollout, or to the introduction of 
key system components? Agencies suggest that a balance is important. Offering 
training prior to the unveiling of a system or component will help build 
excitement and awareness. But, if it is offered too early and trainees have to wait 
a long time before implementation, this excitement could be lost, as could the 
knowledge and skills developed in the training. 

> How will your agency deal with turnover? 

> Beyond initial rollout, how often will PD be offered to users? To keep the staff 
up to speed, training should be ongoing with new-hire training and periodic 
refreshers for experienced staff 

Sustainability 

> How will the PD program be sustained over time? Some education agencies have 
dedicated resources to provide ongoing PD as needed. A train-the-trainer approach 
can establish the capacity at the district and regional levels, allowing agencies to 
continue training over time and expand the training to reach educators and other 
data users. Posting materials, videos, and exercises online can also ensure access 

to training by a broad audience even when more formal PD cannot be provided. 
One state also packaged the materials on a compact disc that was distributed to 
districts, as many had difficulty viewing the material online due to insufficient 
bandwidth. These strategies can be particularly helpful in large states where 
trainees are far apart. Furthermore, as time passes, new staff come on board, and 
the system and its identified uses evolve. A need therefore exists to continually 
gauge user needs and identify areas for refresher courses; more in-depth training; 
or training on new data, tools, and approaches to data use. Help lines can also be 
an effective way of serving users on an ongoing, as-needed basis. 




Tracking 

Recordkeeping serves many purposes. For example, it provides data for audits and 
evaluations, and for fulfilling funding requests based on training and PD efforts. Tracking 
who receives training and PD— in effect, who is taught proper procedures for using the 
data— also offers some level of protection against unethical data use and helps agencies hold 
accountable any end users who might intentionally misuse or manipulate data. 

Logistically, this type of recordkeeping can be approached in several ways. For 
example, training can be required for the authorization of user accounts for analysis and 
querying tools. Registration in training and PD courses might also be tracked via individual 
identifiers maintained in the LDS or staff data system. 

> How will the training and PD offered and delivered be tracked? 

> When was training last offered? 

> What types of training have been most effective (see Efficacy below)? 

> Are more or less individuals being trained over time? 
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W “If it’s worth sending teachers to a workshop, then it’s worth having 
teachers get together to figure out how to put what th^’ve learned 
into practice. Otherwise, if you just leave everybody to their own 
devices, very little of it is going to be implemented.” 

Qolly 2007) 



II 



Efficacy 

> How will PD be measured? 

> What metrics will gauge its success? Counting the number of staff trained is 
not enough. Conducting a survey after the training can help measure trainees’ 
impressions of the program, how their understanding and interpretation of data 
improved, for what purposes they use the data, or how their practices changed 
as a result of the training. Agencies may also gauge the quantitative effects of 
the training on student performance by comparing outcomes before and after 
the training, or by comparing outcomes of students taught by trained and 
untrained teachers. 

Long-term 

> How will continued learning and application of the PD lessons be ensured? 

Onsite data analysts or coaches may help drive local efforts to use the data 
effectively. These coaches may be specially trained in-house staff or external 
consultants. A professional learning community (PLC), or collaborative teacher 
group, can also provide a venue for teachers to practice and hone their skills on 
a continuing basis (see Professional learning communities box below). Some 
agencies have also developed local-level user groups that communicate with 
the state and share knowledge and best practices among districts (DQC 2006). 
Whatever the method, schools should allow time for teachers and administrators 
to review, analyze, and determine how to act on the data. If a culture of effective 
data use is to take root, this time must be built into the schedule rather than being 
an additional burden. 



Professional learning communities 



The term "professional learning community" (PLC) has been applied to 
a range of educator and administrator groups. In this document, a PLC is 
a small, collegial group of educators who work collaboratively to share 
knowledge and learn from one another, develop and improve instructional 
strategies, and increase student learning and performance. 

PLCs are usually established at the school level, within grades and subject 
areas. They may be teacher-driven, but should have principal buy-in to 
ensure members have meeting time and other needed support. Social 
networking tools have allowed these groups to easily communicate with one 
another across schools and districts (see A Case Study: Soaring to Western Heights). These 
groups can provide an effective venue for improving data use by helping educators analyze, 
interpret, and mobilize based on facts about student and teacher needs. 
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T he Oregon Department of Education, with several partner 
organizations, embarked on a statewide professional 
development (PD) program called the Direct Access To 
Achievement (DATA) Project. To meet the needs of educators 
in the field, the first year was spent assessing needs through 
focus groups around the state and written surveys of principals, 
teachers, superintendents, IT staff, and other stakeholders. Project 
leaders sought to answer the core question: "What do we need to 



T 



improve the use of data to improve student achievement in Oregon?" Not only did they receive some resounding 
responses, the project very quickly gained grassroots interest and support. Based on the findings, a curriculum 
was developed. After a pilot round of training was complete, the "data use for instructional improvement" arm 
of the project was adjusted based on trainee feedback. It was then rolled out statewide in 2008, when trainers 
scheduled and ran seminars across the state for school and district staff. 

Data use for instructional improvement: Training framework 

j Instructional training for the Oregon DATA Project is currently composed of four training strands. 

Creating a Data Culture: One of Inquiry 

* This two-day course for lead and master teachers, superintendents, directors, and 
principals is meant to plant the seed for a data culture — that is, one that recognizes data as an integral 
part of teaching, learning, and school management. In this seminar, school leaders examine the story 
behind the numbers and learn to proactively respond to their new insights. 

Using Data to Improve Learning in Districts and Schools 

* This interactive two-day seminar for school leaders such as directors, principals, and lead 
teachers offers in-depth analysis activities and strategies. Participants examine data to make leadership 
and instructional choices within the framework of a data-driven decisionmaking process. Participants 
examine their own, real data on their own, real students so that the process is experienced and 
understood in context. By the end of the seminar, participants have learned the methodology and 
made concrete improvement decisions about their district or school. 

Using Data to Improve Learning in the Ciassroom 

Focus in this two-day seminar is narrowed to the classroom level to help teachers and 
administrators use data to improve individual student learning. Training offers a chance to evaluate 
common assessments used in Oregon, as well as create an assessment framework and assessments 
aligned to instructional goals, including content, skills, and cognitive demand. Sessions include in-depth 
training on using formative and summative data to improve both classroom- and student-level outcomes; 
practical instruction on how to organize, analyze, and tie data back to improved instruction; and tools for 
progress monitoring. 
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Essential Skills in Reading: Impact on Teaching and Learning 

* This three-part strand was designed to help district teams create a K-1 2 system to address 
the skills deemed critical tor students' future success. Teams are taught to use benchmark tests, progress 
monitoring, and scoring guides. Training offers instruction on creating an assessment and instruction 
system based on the Common Core State Standards (CCSS) (http://www.corestandards.org) and 
research-based practices. Participants also learn how to determine student progress toward meeting the 
benchmarks; discuss implementation of the CCSS; adjust and target student instruction; and create a 
vehicle for meaningful reflection and team evaluation. 






When designing a statewide PD program on data use for instructional improvement, planners took into account 
the considerable variation in technologies used for data storage and analysis across the state. Six discrete data 
warehouses in the state can transfer data to a central state data warehouse for reporting, research, or sharing. 
These separate systems have administrator dashboards, though only some equip teachers. To deal with these 
differences, Oregon's training was designed to provide a higher level discussion of data use, laying out guiding 
principles to help staff analyze information regardless of the technology available. For example, participants 
gain an understanding of key data and data use terminology to establish a common language, including abstract 
concepts. Trainees are taught to locate, analyze, and tie data back to instruction and inform improvement plans; 
and to assess the alignment of district and school improvement plans, curricula, and programs. They also learn to 
evaluate programs based on merit— how well the programs are achieving their intended, measurable goals. 

While the main phase of Oregon's training has ended, sustainability has been, and continues to be, a central 
goal. Several approaches have been taken to maintain the training effort: 

■ All training materials and full video segments from the seminars have been posted online and distributed 
widely as DVDs. 

■ DATA certified over 1 00 trainers around the state. 

■ Leaders partnered with regional Education Service Districts (ESDs) to carry on training at the district and 
regional levels. 

■ Several districts decided to build the capacity to hold training sessions locally. 

■ The project staff teamed up with postsecondary partners to develop a more in-depth curriculum that will 
eventually be offered at several state universities as part of pre-service training. 

■ To create further incentive, administrators and teachers can earn credit toward licensure by attending 
training. 

To gauge the project's effectiveness, the impact on student performance has been measured; early data suggest 
that students of pilot district trainees are performing significantly better since training took place. In addition, 
teacher data-use skills have been assessed (trainees are asked to respond to a hypothetical scenario using data), 
and staff opinions and beliefs about data use have been collected; for example, are assessments an unnecessary 
burden or a valuable tool tor improving instruction and student outcomes. 

Oregon's data use project has gained national attention and several other states are currently considering 
using DATA as a model for their own PD initiatives. For more information, including training session videos and 
supporting materials, visit the project website at h1tp://www.oregondataproject.org. 
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Chapter 5 



Conclusion 




' hen the vision for this guide series was crafted, states and districts around 
I the country were already in various stages of LDS development. Some had 
had them in place for many years and were striving to expand their utility, others were just 
beginning to find their way. Since then, education agencies continued to make great strides 
thanks to committed state and district leaders, the federal government, and a number of 
nonprofit organizations and service providers. 



An Impetus for Innovation in Education Data 

In recent years, the U.S. Department of Education has elevated data to a new level of 
importance, tying LDS development and innovative data use to competitive funding 
opportunities via the Institute of Education Sciences (lES) Statewide Longitudinal Data 
System (SLDS) Grant Program and the Race to the Top. Indeed, the message from the 
the federal government on education data has been getting clearer and more resounding: 
In order to realize a more effective, efficient, and equitable education system, education 
stakeholders need detailed, accurate data to provide objective facts about what is 
happening in classrooms, schools, districts, and states. 

The thrust has been strong and the progress made towards establishing robust LDSs 
has been remarkable, exceeding the expectations of many. However, education agencies 
vary widely on many key LDS features and capabilities, including student and staff ID 
systems that link data about students and their teachers, the expansion of data access, the 
implementation of user-friendly analytical and querying tools, adherence to common data 
standards, and the integration and sharing of data across state lines and through the P-20 
pipeline. And despite tremendous advancement and innovation at both the state and local 
levels, much work still remains. 
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An Eye Toward the Future 

As they continue to develop longitudinal data systems and break down the barriers that 
once prevented information sharing and access, education agencies are making valuable, 
detailed facts available to users. How to most effectively and fairly use those data is 
still being debated across the country, and stakeholders— from state and local leaders to 
researchers— are exploring different approaches to leveraging the information. While some 
of these uses have been met with controversy, the possibilities and capacity to utilize this 
information will only grow. Great insights and improvements in policy and practice will 
be made but, at the same time, users will have to be careful in their interpretations and 
applications of the information. Numbers can explain a lot, but they cannot tell the 
whole story. 

No doubt, the road ahead will be filled with obstacles and setbacks. Data quality, 
detail, linkages, access, privacy, and use will likely remain persistent challenges as education 
agencies continue to develop and enhance their LDSs. However, the remarkable progress 
made over the past several years is extremely encouraging. As more progress is made, 
these data will provide ever greater insight into the experiences of students as they journey 
through the education system and out into the workforce— a clear view of reality that has 
been blurred for too long. This is the kind of travel through time that LDSs are making 
possible. This is what makes all the effort worthwhile. 
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Appendix A 



Overview oe the LDS 
Guide Series 



The Traveling Through Time: The Forum Guide to Longitudinal Data Systems series consists of 
four guides intended to help state and local education agencies meet the many challenges 
involved in developing a robust longitudinal data system (LDS), populating them with 
quality data, and using this new information to improve the education system. 

Book One of Four: What is an LDS? focuses on the fundamental questions of what an 
LDS is (and what it is not), what steps should be taken to achieve a sound and successful 
system, what components make up an ideal system, and why such a system is of value 
to education. Book Two of Four: Planning and Developing an LDS, discusses the critical 
planning and development phases of an LDS project, from stakeholder engagement and 
needs assessment all the way through to system evaluation. Book Three of Four: Effectively 
Managing LDS Data explores several fundamental challenges of data management, 
focusing on data governance, data quality, privacy, and security. Finally, Book Four of 
Four: Advanced LDS Usage, addresses the effective utilization of LDS data, discussing the 
users and uses of the data; and emphasizes the need for effective training and professional 
development. The figure on the next page lays out the major issues covered in each of the 
four books in this Fomm guide series. 

> Book One: What is an LDS? is available online at: 
http://nces. ed.gov/forum/pub_201 0805. asp 

> Book Two: Planning and Developing an LDS is available online at: 
http://nces. ed.gov/forum/pub_201 1804. asp 

> Book Three: Effectively ManagingLDS Data 'is available online at: 
http://nces. ed.gov/forum/pub_201 1805. asp 

To order a print copy of any of these guides, please contact: 

ED Pubs 

U.S. Department of Education 
P.O. Box 22207 
Alexandria, VA 22304 

Or call toll free 1-877-4ED-PUBS or order online at http://www.edpuhs.gov 
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Book hWhot Is on LDS? 



■ Understanding what an LDS is (and is not) 

■ Appreciating the organizational steps needed to 
institute and effectively use an LDS 

■ Identifying the technical features and 
capabilities of an effective LDS and the 
additional features that can enhance the 
system's utility 

■ Recognizing the benefits of an LDS 



Book II: Planning and 
Developing on LDS 

■ Engaging stakeholders 

■ Describing the current system 

■ Envisioning the desired system 

■ Defining needs, including data and functionality 

■ Gaining buy-in and funding 

■ Building relationships 

■ Writing an RFP 

■ Building or buying a system or components 

■ Transferring knowledge (e.g., from developers 
to staff) 

■ Defining and measuring success 

■ Refining the system 



Book III: Effectively 
Managing LDS Data 



Book IV: Advanced 
LDS Usage 



■ Defining governance structure 

■ Defining roles and responsibilities 

■ Collaborating to improve data quality and 
streamline operations 

■ Managing changes to the system 

■ Training staff to ensure data quality 

■ Auditing/validating data at all levels 

■ Establishing/following data standards 

■ Securing data to protect privacy 

■ Providing users access to key data 



■ Collecting, storing, and delivering key data 

■ Developing useful reports to fulfill common 
data requests and needs 

■ Developing user-friendly data tools to facilitate 
access and analysis 

■ Training users to utilize the technology 

■ Building awareness, understanding, and 
analytical capacity 
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Appendix C 

Additional Resources 



Chapter 1 . Data Use: Where It Has Been, Where It Is Going 

Information Won’t Be Used If No One Can See It: Why States 
Should Publish Statistics Based on Longitudinal Student Data 

Data Quality Campaign (2008). 

http://www.dataqualitycampaign.org/files/publication-information_wont_be_used_if_no_one_ 
sees_it- 0901 08.pdf 

This article discusses the need get the right data to users, ensure the data are easily 
accessible, and make sure users know how to interpret the data. 

The New Stupid 

American Enterprise Institute for Public Policy Research (2008). 
http://www. aei. org/article/289 72 

This article warns that, while data and research are powerful tools that can be used to 
improve education, we should use and interpret them with caution. 

Chapter 2. Prerequisites for Effective Data Use 

State Education Data Systems That Increase Learning and 
Improve Accountability 

Learning Point Associates (2004). 

http://dataqualitycampaign.org/files/Tools-State_Education_Data_Systems_ThatJncrease_ 

Learning_and_Improve_Accountability.pdf 

This paper examines the components needed to address system upgrades and provides 
policy recommendations to help states create data systems that promote accountability and 
improve student learning. 

Addressing the Need for Better Data on Teaching in Colorado: 
Unique Teacher Identifier Stakeholder Process Report 

The Alliance for Quality Teaching (2007). 
http://aqt.civicore.com/modules/resources/resources/69.pdf 

This report provides information about unique teacher identifier systems, and disseminates 
the outcomes of the four stakeholder meetings discussing such a system for Colorado. 
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Telling Compelling Stories With Numbers 

http://nces. ed.gov/programs/slds/nov08 _j>resentations. asp 

Session VI, a presentation on making sense of data, is relevant for education agency staff 
who want to know more about data use in presentations. Variations in data presentation 
through graphs and figures are presented, as are a definition and discussion of dashboards. 

Chapter 3. What Do Stakeholders Get? LDS Data Uses and Users 

Data Use 

The University of Texas at Austin 
http://edadmin. edb. utexas. edu/datause/index. htm 

This website includes many publications related to the use of data for informed 
decisionmaking at the local level. 

What Is the Role of the State Education Agency in Providing Local 
Education Agencies With Access to Key Data for Instruction? 

Education Information Management Advisory Consortium (2007). 

This brief summarizes points raised in an EIMAC roundtable; participants discussed ways 
state education agencies could provide useful longitudinal data to districts. 

Protecting Student Records and Facilitating Education 
Research: A Workshop Summary 

National Research Council (2008). 
hUp-./Zimsm. nap. edu/catalog.php ?record_id=l 2514 

Chapter 3 of this report discusses the benefits of research using longitudinal data. 

The Next Step: Using Longitudinal Data Systems to Improve 
Student Success 

Data Quality Campaign (DQC 2009). 

http://dataqualitycampaign.org/files/NextStep.pdf 

This report discusses a number of key steps that education agencies need to take in order to 
facilitate effective data use. 

National Center for Analysis of Longitudinal Data in 
Education Research 

http://www. caldercenter. org 

Visit the National Center for Analysis of Longitudinal Data in Education Research 
(CALDER) for examples of research made possible by the availability of longitudinal 
student data. In addition to a large collection of reports that analyze longitudinal data, 
the site provides links to several longitudinal state data sets (http://'ww'w. caldercenter.org/ 
research /statedata. cfm ) . 
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Achieving With Data: How High-Performing School Systems Use 
Data to Improve Instruction for Elementary Students 

Center on Educational Governance, University of Southern California (2007). 

http://nces.ed.gov/programs/slds/ldsshare/documentdirectory/dataaccessuse/datause/ 

achievingwithdata.pdf 

This report looks at several school systems that have successfully used data to improve 
student learning. The document highlights their experiences and suggests key strategies to 
help other districts not only become more data driven, but also use the data effectively. 

Implementing Data-Informed Decision Making in Schools— 
Teacher Access, Supports, and Use 

U.S. Department of Education (2009). 

http://www.ed.gov/rschstat/eval/tech/data-informed-decision/data-informed-decision.doc 

This report presents findings from several studies on data-informed decisionmaking in 
schools. It draws on surveys, interviews, and case studies to explore the prevalence and 
character of data use in schools, including data availability, use, and understanding. The 
support provided to help educators effectively use the data to improve achievement is also 
considered. 

A Policymaker’s Guide to the Value of Longitudinal Student Data 

Education Commission of the States (2002). 
http://www. ecs. org/clearinghouse/40/2 1/402 l.htm 

This early report addresses the types of policy questions longitudinal student data can answer. 

Data Mining and Statistics: What Is the Connection? 

The Data Administration Newsletter (2004). 
http://www. tdan. com/view-articles/ 522 6 

This article describes statistics and data mining, explores the relationship between the two fields 
of inquiry, and argues that both approaches are valuable in efforts to understanding data. 

Teachers’ Use of Student Data Systems to Improve Instruction 
U.S. Department of Education (2007). 

http://www2.ed.gov/rschstat/eval/tech/teachers-data-use/teachers-data-use-intro.html 

Using national surveys of teachers and school districts, this report provides the first 
national estimates of K-12 teachers’ access to, and use of, electronic student data 
management systems. 

Making Sense of Data-Driven Decision Making in Education: 
Evidence From Recent RAND Research 

RAND Corporation (2006), 

http://www.rand.org/pubs/occasional _p>apers/OPl 70 

This paper discusses research on how schools and districts are analyzing a range of data, 
including but not limited to achievement test results to inform decisionmaking to improve 
student outcomes. 
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Tapping into the Power of Longitudinal Data: A Guide for 
School Leaders 

Data Quality Campaign (DQC 2008). 

http://dataqualitycampaign.org/files/publications-tapping_into _the _j?ower_ofJongitudinal_data-a_ 
guide _for_schooljeaders-01 01 08.pdf 

This guide looks at ways teachers and principals can use longitudinal data to meet students’ 
individual needs and improve performance. 

Linking Teacher and Student Data to Improve Teacher and 
Teaching Quality 

Data Quality Campaign (DQC 2007). 

http://dataqualitycampaign. org/files/Meetings-DOC Quarterly Issue B rief_03 1 207.pdf 

This brief discusses the benefits of linking student and teacher data, important metrics 
for improving teacher quality and retention, how to develop comprehensive teacher data 
systems, the pros and cons of value-added measurements, and more. 

Involving Teachers in Data-Driven Decision Making: Using 
Computer Data Systems to Support Teacher Inquiry and Reflection 

Wayman,J. {1005). Journal of Education for Students Placed at Risk, 10{3), 295-308. 

http://edadmin. edb. utexas. edu/datause/papers/Wayman %2 0- %2 0 Teacher%2 OData %2 0 Use. pdf 
This article discusses teacher use of data systems and tools, providing insight into their 
function and discussing conditions that make them most useful to teachers. 

What Works Clearinghouse (WWC) 

Institute of Education Sciences. 
http://ies. ed.gov/ncee/wwc 

This website provides educators, policymakers, researchers, and the public with scientific 
evidence for what works in education. This central, trusted source promotes informed 
education decisionmaking through easily accessible databases and user-friendly reports on 
the effectiveness of replicable education interventions (programs, products, practices, and 
policies) intended to improve student outcomes. 

Getting the Evidence for Evidence-Based Initiatives: How the 
Midwest States Use Data Systems to Improve Education Processes 
AND Outcomes 

National Center for Education Evaluation and Regional Assistance (2007). Issues andAnswers 
Report (REL 2007-016). U.S. Department of Education, Institute of Education Sciences. 

http://ies.ed.gOv/ncee/edlabs/regions/midwest/pdf/rel_2007016_sum.pdf 

This report reviews the progress of several midwestern states in developing LDSs and the 
use of data systems in general. Based on interviews with SEA officials and federal agency 
staff, the authors review the work that was done, the challenges that were faced, and the 
current requirements being pursued by the states. 
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Continuous Improvement: It Takes More Than Test Scores 

Bernhardt, V. (2004). ASCA Leadership Magazine, Nov.-Dee. 2004. 

http://eff.csuchico.edu/downloads/testscores.pdf 

This article summarizes why analyzing state assessment results is only the beginning of 
effective data-driven decisionmaking. 

Intersections: New Routes Open When One Type of Data 
Crosses Another 

Bernhardt, V. Journal of Staff Development, 2/(1), 33-36. 

http://eff.csuchico.edu/downloads/intersct.pdf 

This article discusses how different types of data can be used to help educators answer key 
questions about student learning and the effectiveness of educational inputs. 

Multiple Measures 

Bernhardt, V. (1998). California Association for Supervision and Curriculum Development. 
http://eff.csuchico.edu/downloads/mmeasure.pdf 

This article emphasizes the need to consider a range of data about education in addition 
to student assessment data, commonly the sole source of information available. The 
author suggests we need other types of education data, including student demographics, 
educational inputs, and perceptions. The article describes why each type of data is useful, 
and what can be learned by using a combination of these data to gain greater insight before 
making decisions. 

Leadership Summit 2006 Toolkit: Using Data for School Reform 

State Educational Technology Directors Association (2006). 
http://www. setda. org/ toolkit/ nlitoolkitZ 006/ data/ index.htm 

This toolkit offers practical checklists, matrices, survey instmments, sample letters, and 
other tangible components that can help education technology leaders at all levels to 
improve teaching and learning at the school and student levels. 

Benefits of and Lessons Learned From Linking Teacher and 
Student Data 

Data Quality Campaign (DQC 2007). 

http://www. dataquality campaign. org/files/puhlications-henefits_pf_andJessonsJeamed Jrom_ 
linking_teacher_and_student_data-120607.pdf 

This brief looks at how several states are linking student and teacher data, the benefits of 
those links, and lessons learned. 

Using Data to Impact Instruction 

Aldine Independent School District (2006). 

http://dataqualitycampaign.org/files/Presentations-DQC_Quaterly_Meeting_Using_Data_to_ 

Imp act instruction _092 5 06.pdf 

This presentation provides an overview of the Aldine Independent School District’s 
approach to data use for instructional planning. The Texan district uses a combination of 
trailing and leading data, creates action plans based on those data, and keeps scorecards to 
track progress at various levels of the organization. 
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Research Partnerships and Data Sharing Agreements 

Protecting Student Records and Facilitating Education 
Research: A Workshop Summary 

National Research Council (2008). 

http: //books, nap. edu/openbook.php ?record_id=l 25 14/S^page=44 

See pages 45-46 and 78-79 for practical discussions of research partnerships and data- 
sharing. 

lES Statewide Longitudinal Grant Program’s LDS Share 

Institute of Education Sciences. 

http://nces. ed.gov/programs/slds/ldsshare/slds. aspx 

This document-sharing resource contains many sample MOUs submitted by education 
agencies. 

State Educational Agencies Form Partnership to Make P-20 Data 
System Information Available for Reporting, Analysis, and Research 

Kentucky Education and Workforce Development Cabinet (2009). 
http://educationcabinet.ky.gov/newsroom/pressreleases/stateeducationalpartnership.htm 

This press release discusses Kentucky’s work to establish data sharing across P-20 through 
collaboration between K-12 and postsecondary education organizations, as well as an 
educator workforce development body. The partnership allows data sharing and analyses 
across these sectors. 

Balanced Assessment System 

Professional Papers and Recent Staff Articles 

Pearson Assessment Training Institute. 
http://www. assessmentinst. com/resources/papers 

This page links to a large collection of papers on balanced assessment and related topics. 
Assessment for Learning Defined (Stiggins 2005) is especially useful. 

Data-Driven Teaching: Tools and Trends 

Rennie Center for Education Research and Policy (2006). 
http://www. renniecenter. org/0224 _Brief_FINAL.pdf 

This brief evaluates the implementation of three formative assessments in urban school 
districts. It offers useful information for selecting a formative assessment system that will 
meet a district’s needs; and for effectively implementing the system, and then applying 
those formative assessment data. 
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Growth Measures and Models 

Are We There Yet? What Policymakers Can Learn From 
Tennessee’s Growth Model 

Barone, C. (2009). 

http://w'ww. educationsector. org/research/research_show. htm ?doc_id=84705 8 

This article evaluates the Tennessee Growth Model, pointing out its strengths and 
weaknesses. The page also links to a response from the model’s developer. 

A “Student-Centered” Growth Model for School 
Accountability 

Oregon Department of Education (2009). 

http://w'ww. ode. state, or. us/wma/policy/accountability/gro'wth/growth-overview.pdf 

This document provides a brief summary of Oregon’s growth model, including a short 
description, explanation of why the state and its stakeholders took this approach to 
measuring student academic growth, and links to a file that includes a growth-target 
calculator with formulas. 

Value-Added Models 

Separating Growth From Value-Added 

Yeagley, R. (2007). The School Administrator, 64(1):18. 

http://w'ww.thefreelibraTy.com/separating+growth+from+value+added:+two+academic-\-models+ 
offer. ..-aOl 57588355 

This article points out the differences between approaches to measuring value-added 
and growth and discusses the merits and uses of each. Another article follows this one 
addressing the implementation of a growth model in a California school system. 

Comparisons Among Various Educational Assessment 
Value-Added Models 

SAS Institute (2006). 

http://nces.ed.gov/programs/slds/ldsshare/documentdirectory/dataaccessuse/valueaddedmodels/ 

billsanders_valueaddedmodelcomparisons.pdf 

This white paper highlights the differences between several classroom-level, value-added 
models; and discusses the advantages and disadvantages of each, especially as they relate to 
teacher impact. 

Value-Added Indicators: A Powerful Tool for Evaluating 
Science and Mathematics Programs and Policies 

Meyer, R. (2000). National Center for Improving Science Education. 
http://addingvalue. wceruw. org/related%2 Obibliography /articles/my er. pdf 

This brief highlights the advantages of value-added models over the more commonly used 
indicators of student performance: average and median test scores and proficiency-level 
indicators. 
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Value-Added Indicators: Do They Make an Important Difference? 
Evidence From the Milwaukee Public Schools 

Meyer, R. (2003). Wisconsin Center for Education Research. 
http://www. eric. ed.gov/PDFS/ED49 7583.pdf 

This working paper demonstrates the utility of value-added data using a Wisconsin 
district’s data. It discusses why this method of measuring academic performance is superior 
to other more commonly used indicators. 

Methodological Concerns About the Education Value-Added 
Assessment System 

Amrein-Beardsley, A. Educational Researcher 37(2), 65-75. 

http://www. aera. net/uploadedfiles/publications/journals/educational_researcher/3 702/ 
03edr08_065-075.pdf 

This article examines the Education Value-Added Assessment System (EVAAS), focusing 
on the methodological and practical issues associated with its approach. 

Value-Added Analysis and Education Policy 

National Center for Analysis of Longitudinal Data in Education Research (2007). 
http://imsm. urban.org/uploadedpdf/41 1 577 _value-added_analysis.pdf 

This brief describes issues related to estimating and measuring the quality of instruction, 
highlighting the importance of taking student differences into account and the advantages 
of using academic growth rather than differences in test scores. 



Value-Added Models in the News... 

Growth Data for Teachers Under Review 

Education Week 28(f), October 2008. 

http:/ /-WWW. edweek. org/ew/articles/2 008/1 0/22/09value_ep. h2 8. html?tmp=l 64757091 

This article summarizes the issues and challenges around the use of value-added data for 
teacher evaluations. 

“Value-Added” Gauge of Teachers Probed 

Education Week28(f6),]\Ay 1009. 

This article discusses several studies that evaluated value-added measures. It provides a 
good summary of the contemporary arguments around the use of these measures. 

Scrutiny Heightens for “Value Added” Research Methods 

Education Week27{56), May 2008. 

This article makes cautionary points about the use of value-added measures. While they 
hold great promise, the article suggests that practitioners must proceed carefully as we learn 
more about how best to model and use these measures. 
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Back to School: Performance Data Driving Education Now 

Pittsburgh Post-Gazette, August 29, 2006. 

http://www.post-gazette.com/pg/06241/71 71 04-298. stm 

This article discusses Pennsylvania’s plan to implement a value-added system across the 
state, giving every district access to its results. 

William Sanders Describes Value-Added Assessment for 
Measuring Student Progress 

University of North Carolina School of Education (2006). 
http://soe. unc. edu/nems_events/nems/2 006/sanders_william.php 

This story reports on a lecture by William Sanders on his value-added assessment system. 

Chapter 4. Professional Development and Training for 
Effective Use of IDS Data 

Direct Access to Achievement 

Oregon Department of Education. 
http://www. oregondataproject. org/ 

This website on Oregon’s Direct Access to Achievement (DATA) project contains a host of 
resources including training materials for instructional improvement informed by data. The 
site also includes a strand on improving data quality. 

Beyond the Numbers: Making Data Work for Teachers and 
School Leaders 

White, S. (2005). Advanced Learning Press. 

This book on local-level, data-driven decisionmaking was used in Oregon’s professional 
development initiative, the DATA Project. In addition to discussing the foundational 
principles for data analysis and the strategies necessary to turn analysis into action, it 
provides a specific framework for analyzing a range of data, yet is general enough to 
support data analysis efforts regardless of the specific technology available to users. 

Technology @ Your Fingertips 

National Forum on Education Statistics (1998). 
http://nces. ed.gov/pubs98/982 93rev.pdf 

See chapter 6 for a collection of best practices on training staff and others to use 
technology. While the chapter applies to general technology solutions, rather than specific 
to data systems, many of its lessons are relevant to an EDS training effort. 

Collaborative Approaches to Education Data Training and 
Professional Development 

Center for Educational Performance and Information (2009). 
http://nces.ed.gOv/programs/slds/zip/08 J^_09a.zip 

This presentation provides an overview of Michigan’s approach to professional 
development on data quality and data use. 



Book Four of Four: Advanced IDS Usage 



61 




Data Wise: A Step-by-Step Guide to Using Assessment Results to 
Improve Teaching and Learning 

Harvard Educational Pub Group (2005). 

This book discusses what school leaders need to know and do to ensure that the piles of 
student assessment results landing on their desks are used to improve student learning in 
their schools. 

Making Sense of Data-Driven Decision Making in Education: 
Evidence From Recent RAND Research 

RAND Corporation (2006). 

http ://'www. rand.org/pubs/occasional _j>apers/OPl 70 

See pages 7-8 for a discussion and analysis of professional development approaches. 

Teaching and Learning Cooperative: Sample Activities 

Poway Unified School District. 

http://www.powayusd.com/projects/litera(y/ssttl/pdfs/part%20ii%20samples.pdf 

This document describes various approaches to local-level professional development. 
While not specific to data analysis, it includes a section on teacher research and guides 
educators through an exercise of data-driven decisionmaking. 
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Appendix D 

Forum AND Other NCES 
Resources 



The Forum Guide to Data Ethics (NFES 2010-801) 

http://nces. ed.gov/forum/pub^ 01 0801. asp 

Every day, educators collect and use data about students, staff, and 
schools. Some of these data originate in individual student and staff 
records that are confidential or otherwise sensitive. And even those 
data that are a matter of public record, such as aggregate school 
enrollment, need to be accessed, presented, and used in an ethically 
responsible manner. While laws set the legal parameters that govern 
data use, ethics establish fundamental principles of “right and 
wrong” that are critical to the appropriate management and use of education data in the 
technology age. This guide reflects the experience and judgment of experienced data 
managers; while there is no mandate to follow these principles, the authors hope that the 
contents will prove a useful reference to others in their work. 




Crisis Data Management: A Forum Guide to Collecting and 
Managing Data about Displaced Students (NFES 2010-804) 

http://nces. ed.gov/forum/pub_201 0804. asp 

This publication provides guidelines that can be used by elementary 
and secondary education agencies to establish policies and procedures 
for collecting and managing education data before, during, and after a 
crisis. 



The Forum Guide to Metadata: The Meaning Behind Education 
Data (NFES 2009-805) 

hup ://nces. ed.gov /forum/pub _2009805. asp 
This guide offers best practice concepts, definitions, 
implementation strategies, and templates/tools for an audience of 
data, technology, and program staff in state and local education 
agencies. This resource was developed to improve these audiences’ 
awareness and understanding of metadata and, subsequently, the 
quality of the data in the systems they maintain. 
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Every School Day Counts: Forum Guide to Collecting and 
Using Attendance Data (NFES 2009-804) 

http://nces. ed.gov /forum/pub_2009804. asp 

This guide offers best practice suggestions on collecting and using 
student attendance data to improve performance. It includes a 
standard set of codes to make attendance data comparable across 
districts and states. The publication includes real-life examples of how 
attendance information has been used by school districts. 




NCES Handbooks and NCES Handbooks Online 

http://www.nces.ed.gov/programs/handbook 

Hfend300(<5 ^line — — NCES Handbooks are a valuable source of metadata 
for organizations and individuals interested in education data. These print and online 
resources define standard education terms for students, staff, schools, local education 
agencies (LEA), intermediate education agencies, and state education agencies (SEA). The 
Handbooks are intended as reference documents for public and private organizations, 
including education institutions and early childhood centers; as well as education 
researchers and other users of education data. In order to improve access to this valuable 
resource, NCES has also developed the NCES Handbooks Online, a web-based tool that 
allows users to view and download information via an electronic table of contents, a drill- 
down finder, element name and first letter searches, and advanced query options. 



National Education Data Model 

tnudiTu ri*n MriHi http://nces.ed.gOv/forum/datamodel/index.aspx 

The National Education Data Model (NEDM) is the first 
non-proprietary, national education data model developed to help schools, LEAs, and 
states design or guide the selection of systems for instmctional delivery, data-driven 
decisionmaking, data collection, operations, and reporting. The model provides a national 
blueprint to help schools evaluate and improve instructional tools; communicate needs to 
their umbrella agency or to vendors; enhance the movement of student information from 
one LEA to another; and, in the end, have better tools to inform instruction. NEDM can 
be used by educators, vendors, and researchers to understand the information required for 
teaching, learning, and administrative systems. 

Identity Crisis: Forum Guide to Implementing New 
and Ethnicity Categories (NFES 2008-802) 

http://nces.ed.gOv/forum/pub_2008802.asp 
This best-practice guide was developed to help state and local 
education agencies implement the new federal race and ethnicity 
categories, thereby reducing redundant efforts within and across states, 
improving data comparability, and minimizing reporting burden. 

Users may select and adopt strategies that will help them quickly begin 
the process of implementation in their agencies. 



Managing an 
Federal Race 
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Forum Guide to Core Finance Data Eiements (NFES 2007-801) 

http://nces.ed.gOv/forum/pub_2007801.asp 
This publication establishes current and consistent terms and 
definitions for maintaining, collecting, reporting, and exchanging 
comparable information related to education finances. It is designed 
to accompany Financial Accounting for Local and State School 
Systems: 2003 Edition by identifying common reporting requirements 
and defining frequently used indicators and calculations that use data 
elements from accounting and other data systems. 

Forum Curriculum for Improving Education Data: A Resource for 
Local Education Agencies (NFES 2007-808) 

http://nces. ed.gov/forum/pub_2 007808. asp 

This resource supports efforts to improve the quality of education 
data by serving as training materials for K-12 school and district staff. 
It provides lesson plans, instructional handouts, and other resources; 
and presents concepts necessary to help schools develop a culture for 
improving data quality. 





Forum Guide to Decision Support Systems: A Resource for 
Educators (NFES 2006-807) 

http://nces.ed.gOv/forum/pub_2006807.asp 

This Forum guide was developed to help the education community 
better understand what decision support systems are, how they are 
configured, how they operate, and how they might be developed and 
implemented in an education setting. 




Forum Guide to Virtual Education (NFES 2006-803) 

■ http://nces.ed.gOv/forum/pub_2006803.asp 

This publication offers recommendations for collecting accurate, 
comparable, and useful data about virtual education in elementary and 
secondary education settings. It highlights policy questions and data 
elements critical to meeting the information needs of policymakers, 
administrators, instmctors, and parents of students involved in virtual 
education. 
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Forum Guide to the Privacy of Student Information: 

A Resource for Schools (NFES 2006-805) 

http://nces.ed.gOv/forum/pub_2006805.asp 

This publication was written to help school and local education agency 
staff better understand and apply the Family Educational Rights and 
Privacy Act (FERPA), a federal law that protects the privacy interests 
of parents and students with respect to information maintained in 
student education records. It defines terms such as “education records” 
and “directory information,” and offers guidance for developing 
appropriate privacy policies and information disclosure procedures 
related to military recmiting, parental rights and annual notification, videotaping, online 
information, media releases, surveillance cameras, and health-related information. 

Accounting for Every Student: A Taxonomy for Standard Student 
Exit Codes (NFES 2006-804) 

http://nces. ed.gov /forum/puh_2006804. asp 

This publication was developed to help education agencies develop 
effective information systems for tracking students’ enrollment status. 

It presents a student-level exit code taxonomy for states and districts 
that accounts for 100 percent (not 90 or 110 percent) of all students. It 
also offers “best practice” advice regarding tracking students, collecting 
exit codes data, and distinguishing among high school completion 
credentials. 

to Education Indicators (NFES 2005-802) 

http://nces. ed.gov /forum/puh_2005802. asp 

This publication provides encyclopedia-type entries for 44 commonly 
used education indicators. Each indicator entry includes a definition, 
recommended uses, caveats and cautions, related policy questions, data 
element components, a formula, commonly reported subgroups, and 
display suggestions. The document will help readers better understand 
how to appropriately develop, apply, and interpret commonly used 
education indicators. 

Forum Guide to Building a Cuiture of Quality Data 
(NFES 2005-801) 

http://nces. ed. gov /forum /pub _2005801. asp 

This publication focuses on data entry: getting things done right at the 
source. It recommends a practical process for developing a “Culture 
of Quality Data” based around individual tip sheets for individuals 
involved in providing data, including principals, teachers, office staff, 
school board members, superintendents, data stewards, and technology 
staff 
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Forum Unified Education Technoiogy Suite (2005) 

http://nces. ed.gov/forum/pub_tech_suite. asp 
This publication presents a practical, comprehensive, and tested 
approach to assessing, acquiring, instituting, managing, securing, 
and using technology in education settings. It is written for 
individuals without extensive experience with technology who have been tasked with 
leading technology initiatives in a school or district setting,. 




Forum Guide to Protecting the Privacy of Student Information: 
State and Locai Education Agencies (NCES 2004-330) 

http ://nces. ed.gov/forum/pub_200433 0. asp 

This publication presents a general overview of privacy laws and 
professional practices that apply to information collected for, and 
maintained in, student records. The guide provides an overview of key 
principles and concepts governing student privacy; summarizes federal 
privacy laws; identifies issues concerning the release of information 
to both parents and external organizations; and suggests good data 
management practices for schools, districts, and state education 
agencies. 




Facilities Information Management: A Guide for State and Locai 
Education Agencies (NCES 2003-400) 

http://nces.ed.gOv/forum/puh_2003400.asp 
This publication provides a framework for identifying a basic set 
of school facilities data elements and definitions that will meet the 
information needs of school and community decisionmakers, school 
facility managers, and the general public. It presents recommendations 
for designing and maintaining an information system that addresses the 
condition, design, use, management, and financing of elementary and 
secondary education facilities. Commonly used measures, data elements, 
and additional resources for the practitioner are also included. 

Planning Guide for Maintaining Schooi Facilities 
(NCES 2003-347) 

http://nces. ed.gov/forum/puh_2 003347. asp 

This publication is intended to help school facilities managers plan 
for efficient and effective operations. It provides practical advice on 
a range of topics, including how to conduct a facilities audit, plan for 
maintenance to ensure smooth operations and avoid costly surprises, 
manage staff and contractors, and evaluate maintenance efforts. 
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